| Message ID | 20260312160009.18654-2-david.plowman@raspberrypi.com |
|---|---|
| State | New |
| Headers | show |
| Series |
|
| Related | show |
Hi everyone I'd like to give this topic another prod. I talked about it a bit at Nice, and revisiting this patch seems like a way to start some discussion, though the precise implementation here is more for illustration. One of the principal motivations is for things like burst captures, where you don't want to get held up at the back of the request queue. Does this seem like a reasonable thing to do, or are there better alternatives? Thanks David On Thu, 12 Mar 2026 at 16:00, David Plowman <david.plowman@raspberrypi.com> wrote: > > Add `Camera::queueControls()` whose purpose is to apply controls as > soon as possible, without going through `Request::controls()`. > > A new virtual function `PipelineHandler::queueControlsDevice()` is > provided for pipeline handler to implement fast-tracked application of > controls. If the pipeline handler does not implement that > functionality, or it fails, then a fallback mechanism is used. The > controls will be saved for later, and they will be merged into the > control list of the next available request sent to the pipeline > handler (`Camera::Private::waitingRequests_`). > > This patch is derived directly from Barnabas's previous verion that > implemented the same idea but with a single ControlList, rather than > allowing multiple ControlLists to be queued up for consecutive > frames. > > Signed-off-by: David Plowman <david.plowman@raspberrypi.com> > --- > include/libcamera/camera.h | 1 + > include/libcamera/internal/camera.h | 1 + > include/libcamera/internal/pipeline_handler.h | 7 ++ > src/libcamera/camera.cpp | 61 +++++++++++++++ > src/libcamera/pipeline_handler.cpp | 76 +++++++++++++++++++ > 5 files changed, 146 insertions(+) > > diff --git a/include/libcamera/camera.h b/include/libcamera/camera.h > index b24a2974..93a484e4 100644 > --- a/include/libcamera/camera.h > +++ b/include/libcamera/camera.h > @@ -147,6 +147,7 @@ public: > > std::unique_ptr<Request> createRequest(uint64_t cookie = 0); > int queueRequest(Request *request); > + int queueControls(ControlList &&controls); > > int start(const ControlList *controls = nullptr); > int stop(); > diff --git a/include/libcamera/internal/camera.h b/include/libcamera/internal/camera.h > index 8a2e9ed5..17dda925 100644 > --- a/include/libcamera/internal/camera.h > +++ b/include/libcamera/internal/camera.h > @@ -38,6 +38,7 @@ public: > > std::list<Request *> queuedRequests_; > std::queue<Request *> waitingRequests_; > + std::queue<ControlList> queuedControls_; > ControlInfoMap controlInfo_; > ControlList properties_; > > diff --git a/include/libcamera/internal/pipeline_handler.h b/include/libcamera/internal/pipeline_handler.h > index b4f97477..c25213de 100644 > --- a/include/libcamera/internal/pipeline_handler.h > +++ b/include/libcamera/internal/pipeline_handler.h > @@ -57,6 +57,7 @@ public: > > void registerRequest(Request *request); > void queueRequest(Request *request); > + int queueControls(Camera *camera, ControlList controls); > > bool completeBuffer(Request *request, FrameBuffer *buffer); > void completeRequest(Request *request); > @@ -76,6 +77,12 @@ protected: > unsigned int useCount() const { return useCount_; } > > virtual int queueRequestDevice(Camera *camera, Request *request) = 0; > + > + virtual int queueControlsDevice([[maybe_unused]] Camera *camera, [[maybe_unused]] const ControlList &controls) > + { > + return -EOPNOTSUPP; > + } > + > virtual void stopDevice(Camera *camera) = 0; > > virtual bool acquireDevice(Camera *camera); > diff --git a/src/libcamera/camera.cpp b/src/libcamera/camera.cpp > index f724a1be..f0244707 100644 > --- a/src/libcamera/camera.cpp > +++ b/src/libcamera/camera.cpp > @@ -637,6 +637,16 @@ Camera::Private::~Private() > * queued requests was reached. > */ > > +/** > + * \var Camera::Private::queuedControls_ > + * \brief The queue of pending control lists > + * > + * This queue maintains a list of all the control lists that need to be sent > + * to the pipeline handler with subsequent requests. The top item in the queue > + * will always be sent with the next request going to > + * PipelineHandler::queueRequestDevice(). > + */ > + > /** > * \var Camera::Private::controlInfo_ > * \brief The set of controls supported by the camera > @@ -1378,6 +1388,57 @@ int Camera::queueRequest(Request *request) > return 0; > } > > +/** > + * \brief Queue controls to be applied as soon as possible > + * \param[in] controls The list of controls to queue > + * > + * This function tries to ensure that the controls in \a controls are applied > + * to the camera as soon as possible. If there are still pending controls waiting > + * to be applied (because of previous calls to Camera::queueControls), then > + * these controls will be applied as soon as possible on a frame after those. > + * > + * The exact guarantees are camera dependent, but it is guaranteed that the > + * controls will be applied no later than with the next \ref Request"request" > + * that the application \ref Camera::queueRequest() "queues" (after any requests > + * have been *used up" for sending previously queued controls). > + * > + * \context This function is \threadsafe. It may only be called when the camera > + * is in the Running state as defined in \ref camera_operation. > + * > + * \return 0 on success or a negative error code otherwise > + * \retval -ENODEV The camera has been disconnected from the system > + * \retval -EACCES The camera is not running > + */ > +int Camera::queueControls(ControlList &&controls) > +{ > + Private *const d = _d(); > + > + /* > + * Like requests, controls can't be queued if the camera is not running. > + * Controls can be applied immediately when the camera starts using the > + * Camera::Start method. > + */ > + > + int ret = d->isAccessAllowed(Private::CameraRunning); > + if (ret < 0) > + return ret; > + > + /* > + * We want to be able to queue empty control lists, as this gives a way of > + * forcing another frame with the same controls as last time, before queueing > + * another control list that might change them again. > + */ > + > + patchControlList(controls); > + > + /* > + * \todo Or `ConnectionTypeBlocking` to get the return value? > + */ > + d->pipe_->invokeMethod(&PipelineHandler::queueControls, ConnectionTypeQueued, this, std::move(controls)); > + > + return 0; > +} > + > /** > * \brief Start capture from camera > * \param[in] controls Controls to be applied before starting the Camera > diff --git a/src/libcamera/pipeline_handler.cpp b/src/libcamera/pipeline_handler.cpp > index 5c469e5b..1a87b28c 100644 > --- a/src/libcamera/pipeline_handler.cpp > +++ b/src/libcamera/pipeline_handler.cpp > @@ -398,6 +398,12 @@ void PipelineHandler::stop(Camera *camera) > ASSERT(data->queuedRequests_.empty()); > ASSERT(data->waitingRequests_.empty()); > > + /* > + * Clear out any unapplied controls. If an application wants to be > + * sure controls have been applied, it should wait before stopping. > + */ > + data->queuedControls_ = {}; > + > data->requestSequence_ = 0; > } > > @@ -477,6 +483,49 @@ void PipelineHandler::queueRequest(Request *request) > request->_d()->prepare(300ms); > } > > +/** > + * \brief Queue controls to apply as soon as possible > + * \param[in] camera The camera > + * \param[in] controls The controls to apply > + * > + * This function tries to queue \a controls immediately to the device by > + * calling queueControlsDevice(). If that fails, then a fallback mechanism > + * is used to ensure that \a controls will be merged into the control list > + * of the next available request submitted to the pipeline handler. > + * > + * \context This function is called from the CameraManager thread. > + */ > +int PipelineHandler::queueControls(Camera *camera, ControlList controls) > +{ > + Camera::Private *data = camera->_d(); > + int ret = queueControlsDevice(camera, controls); > + > + /* > + * Don't worry about later request's controls overriding the ones > + * sent here - the application needs to deal with that. > + */ > + > + if (ret == -EOPNOTSUPP) { > + /* > + * Fall back to adding the controls to the next request that enters the > + * pipeline handler. See PipelineHandler::doQueueRequest(). > + */ > + data->queuedControls_.push(std::move(controls)); > + > + /* Counts as "success". */ > + ret = 0; > + > + } else if (ret < 0) { > + /* > + * The pipeline handler is claiming to support queueControlsDevice, > + * but it has failed. This is an error. > + */ > + LOG(Pipeline, Debug) << "Fast tracking controls failed: " << res; > + } > + > + return ret; > +} > + > /** > * \brief Queue one requests to the device > */ > @@ -495,9 +544,21 @@ void PipelineHandler::doQueueRequest(Request *request) > return; > } > > + if (!data->queuedControls_.empty()) { > + /* > + * Note that `ControlList::MergePolicy::KeepExisting` is used. This is > + * needed to ensure that if `request` is newer than pendingControls_, > + * then its controls take precedence. > + */ > + request->controls().merge(data->queuedControls_.front(), > + ControlList::MergePolicy::KeepExisting); > + } > + > int ret = queueRequestDevice(camera, request); > if (ret) > cancelRequest(request); > + else if (!data->queuedControls_.empty()) > + data->queuedControls_.pop(); > } > > /** > @@ -543,6 +604,21 @@ void PipelineHandler::doQueueRequests(Camera *camera) > * \return 0 on success or a negative error code otherwise > */ > > +/** > + * \fn PipelineHandler::queueControlsDevice() > + * \brief Queue controls to be applied as soon as possible > + * \param[in] camera The camera > + * \param[in] controls The controls to apply > + * > + * This function queues \a controls to \a camera so that they can be > + * applied as soon as possible > + * > + * \context This function is called from the CameraManager thread. > + * > + * \return 0 on success or a negative error code otherwise > + * \return -EOPNOTSUPP if fast-tracking controls is not supported > + */ > + > /** > * \brief Complete a buffer for a request > * \param[in] request The request the buffer belongs to > -- > 2.47.3 >
Hi 2026. 07. 23. 18:17 keltezéssel, David Plowman írta: > Hi everyone > > I'd like to give this topic another prod. I talked about it a bit at > Nice, and revisiting this patch seems like a way to start some > discussion, though the precise implementation here is more for > illustration. > > One of the principal motivations is for things like burst captures, > where you don't want to get held up at the back of the request queue. > > Does this seem like a reasonable thing to do, or are there better alternatives? I started typing out a reply in a different thread of yours, but it got too long, so I though it would be more effective to write a quick reply here. I believe there is largely agreement that some kind non-request-submission-tied control setting mechanism is desirable. So I think it would be useful to examine the motivating use cases in a bit more detail. Sorry if this was already done somewhere, I couldn't recall. So the way I understand it, there is a user that is cycling through a set of requests, keeping most of them queued. This works fine, however, sometimes, maybe due to some external event, it wants to capture a number of frames with specific settings. The reason why applying the controls in the subsequent requests is undesirable is because of the delay that the already queued requests introduce. The proposed solution is to have two parallel queues: one for requests, and one for controls. Furthermore, the two are consumed in lockstep. And there is an assumption that the control is queue is mostly empty. This way when the user submits a control list, it can effectively "skip" the requests in the parallel request queue, and arrive near the front of the control queue. And thus it will be "applied to an earlier request", reducing latency. Is this a faithful description? If so, I have two questions: * what if the user wants to enable/disable various streams for the "burst capture"? do you have any thoughts about how that API would work? * is it guaranteed that every control list will be applied to the results of the corresponding request? if not, how would that work with the user enabling/disabling various streams (during the "burst capture")? Thanks, Barnabás Pőcze > > Thanks > > David > > On Thu, 12 Mar 2026 at 16:00, David Plowman > <david.plowman@raspberrypi.com> wrote: >> >> Add `Camera::queueControls()` whose purpose is to apply controls as >> soon as possible, without going through `Request::controls()`. >> >> A new virtual function `PipelineHandler::queueControlsDevice()` is >> provided for pipeline handler to implement fast-tracked application of >> controls. If the pipeline handler does not implement that >> functionality, or it fails, then a fallback mechanism is used. The >> controls will be saved for later, and they will be merged into the >> control list of the next available request sent to the pipeline >> handler (`Camera::Private::waitingRequests_`). >> >> This patch is derived directly from Barnabas's previous verion that >> implemented the same idea but with a single ControlList, rather than >> allowing multiple ControlLists to be queued up for consecutive >> frames. >> >> Signed-off-by: David Plowman <david.plowman@raspberrypi.com> >> --- >> include/libcamera/camera.h | 1 + >> include/libcamera/internal/camera.h | 1 + >> include/libcamera/internal/pipeline_handler.h | 7 ++ >> src/libcamera/camera.cpp | 61 +++++++++++++++ >> src/libcamera/pipeline_handler.cpp | 76 +++++++++++++++++++ >> 5 files changed, 146 insertions(+) >> >> diff --git a/include/libcamera/camera.h b/include/libcamera/camera.h >> index b24a2974..93a484e4 100644 >> --- a/include/libcamera/camera.h >> +++ b/include/libcamera/camera.h >> @@ -147,6 +147,7 @@ public: >> >> std::unique_ptr<Request> createRequest(uint64_t cookie = 0); >> int queueRequest(Request *request); >> + int queueControls(ControlList &&controls); >> >> int start(const ControlList *controls = nullptr); >> int stop(); >> diff --git a/include/libcamera/internal/camera.h b/include/libcamera/internal/camera.h >> index 8a2e9ed5..17dda925 100644 >> --- a/include/libcamera/internal/camera.h >> +++ b/include/libcamera/internal/camera.h >> @@ -38,6 +38,7 @@ public: >> >> std::list<Request *> queuedRequests_; >> std::queue<Request *> waitingRequests_; >> + std::queue<ControlList> queuedControls_; >> ControlInfoMap controlInfo_; >> ControlList properties_; >> >> diff --git a/include/libcamera/internal/pipeline_handler.h b/include/libcamera/internal/pipeline_handler.h >> index b4f97477..c25213de 100644 >> --- a/include/libcamera/internal/pipeline_handler.h >> +++ b/include/libcamera/internal/pipeline_handler.h >> @@ -57,6 +57,7 @@ public: >> >> void registerRequest(Request *request); >> void queueRequest(Request *request); >> + int queueControls(Camera *camera, ControlList controls); >> >> bool completeBuffer(Request *request, FrameBuffer *buffer); >> void completeRequest(Request *request); >> @@ -76,6 +77,12 @@ protected: >> unsigned int useCount() const { return useCount_; } >> >> virtual int queueRequestDevice(Camera *camera, Request *request) = 0; >> + >> + virtual int queueControlsDevice([[maybe_unused]] Camera *camera, [[maybe_unused]] const ControlList &controls) >> + { >> + return -EOPNOTSUPP; >> + } >> + >> virtual void stopDevice(Camera *camera) = 0; >> >> virtual bool acquireDevice(Camera *camera); >> diff --git a/src/libcamera/camera.cpp b/src/libcamera/camera.cpp >> index f724a1be..f0244707 100644 >> --- a/src/libcamera/camera.cpp >> +++ b/src/libcamera/camera.cpp >> @@ -637,6 +637,16 @@ Camera::Private::~Private() >> * queued requests was reached. >> */ >> >> +/** >> + * \var Camera::Private::queuedControls_ >> + * \brief The queue of pending control lists >> + * >> + * This queue maintains a list of all the control lists that need to be sent >> + * to the pipeline handler with subsequent requests. The top item in the queue >> + * will always be sent with the next request going to >> + * PipelineHandler::queueRequestDevice(). >> + */ >> + >> /** >> * \var Camera::Private::controlInfo_ >> * \brief The set of controls supported by the camera >> @@ -1378,6 +1388,57 @@ int Camera::queueRequest(Request *request) >> return 0; >> } >> >> +/** >> + * \brief Queue controls to be applied as soon as possible >> + * \param[in] controls The list of controls to queue >> + * >> + * This function tries to ensure that the controls in \a controls are applied >> + * to the camera as soon as possible. If there are still pending controls waiting >> + * to be applied (because of previous calls to Camera::queueControls), then >> + * these controls will be applied as soon as possible on a frame after those. >> + * >> + * The exact guarantees are camera dependent, but it is guaranteed that the >> + * controls will be applied no later than with the next \ref Request"request" >> + * that the application \ref Camera::queueRequest() "queues" (after any requests >> + * have been *used up" for sending previously queued controls). >> + * >> + * \context This function is \threadsafe. It may only be called when the camera >> + * is in the Running state as defined in \ref camera_operation. >> + * >> + * \return 0 on success or a negative error code otherwise >> + * \retval -ENODEV The camera has been disconnected from the system >> + * \retval -EACCES The camera is not running >> + */ >> +int Camera::queueControls(ControlList &&controls) >> +{ >> + Private *const d = _d(); >> + >> + /* >> + * Like requests, controls can't be queued if the camera is not running. >> + * Controls can be applied immediately when the camera starts using the >> + * Camera::Start method. >> + */ >> + >> + int ret = d->isAccessAllowed(Private::CameraRunning); >> + if (ret < 0) >> + return ret; >> + >> + /* >> + * We want to be able to queue empty control lists, as this gives a way of >> + * forcing another frame with the same controls as last time, before queueing >> + * another control list that might change them again. >> + */ >> + >> + patchControlList(controls); >> + >> + /* >> + * \todo Or `ConnectionTypeBlocking` to get the return value? >> + */ >> + d->pipe_->invokeMethod(&PipelineHandler::queueControls, ConnectionTypeQueued, this, std::move(controls)); >> + >> + return 0; >> +} >> + >> /** >> * \brief Start capture from camera >> * \param[in] controls Controls to be applied before starting the Camera >> diff --git a/src/libcamera/pipeline_handler.cpp b/src/libcamera/pipeline_handler.cpp >> index 5c469e5b..1a87b28c 100644 >> --- a/src/libcamera/pipeline_handler.cpp >> +++ b/src/libcamera/pipeline_handler.cpp >> @@ -398,6 +398,12 @@ void PipelineHandler::stop(Camera *camera) >> ASSERT(data->queuedRequests_.empty()); >> ASSERT(data->waitingRequests_.empty()); >> >> + /* >> + * Clear out any unapplied controls. If an application wants to be >> + * sure controls have been applied, it should wait before stopping. >> + */ >> + data->queuedControls_ = {}; >> + >> data->requestSequence_ = 0; >> } >> >> @@ -477,6 +483,49 @@ void PipelineHandler::queueRequest(Request *request) >> request->_d()->prepare(300ms); >> } >> >> +/** >> + * \brief Queue controls to apply as soon as possible >> + * \param[in] camera The camera >> + * \param[in] controls The controls to apply >> + * >> + * This function tries to queue \a controls immediately to the device by >> + * calling queueControlsDevice(). If that fails, then a fallback mechanism >> + * is used to ensure that \a controls will be merged into the control list >> + * of the next available request submitted to the pipeline handler. >> + * >> + * \context This function is called from the CameraManager thread. >> + */ >> +int PipelineHandler::queueControls(Camera *camera, ControlList controls) >> +{ >> + Camera::Private *data = camera->_d(); >> + int ret = queueControlsDevice(camera, controls); >> + >> + /* >> + * Don't worry about later request's controls overriding the ones >> + * sent here - the application needs to deal with that. >> + */ >> + >> + if (ret == -EOPNOTSUPP) { >> + /* >> + * Fall back to adding the controls to the next request that enters the >> + * pipeline handler. See PipelineHandler::doQueueRequest(). >> + */ >> + data->queuedControls_.push(std::move(controls)); >> + >> + /* Counts as "success". */ >> + ret = 0; >> + >> + } else if (ret < 0) { >> + /* >> + * The pipeline handler is claiming to support queueControlsDevice, >> + * but it has failed. This is an error. >> + */ >> + LOG(Pipeline, Debug) << "Fast tracking controls failed: " << res; >> + } >> + >> + return ret; >> +} >> + >> /** >> * \brief Queue one requests to the device >> */ >> @@ -495,9 +544,21 @@ void PipelineHandler::doQueueRequest(Request *request) >> return; >> } >> >> + if (!data->queuedControls_.empty()) { >> + /* >> + * Note that `ControlList::MergePolicy::KeepExisting` is used. This is >> + * needed to ensure that if `request` is newer than pendingControls_, >> + * then its controls take precedence. >> + */ >> + request->controls().merge(data->queuedControls_.front(), >> + ControlList::MergePolicy::KeepExisting); >> + } >> + >> int ret = queueRequestDevice(camera, request); >> if (ret) >> cancelRequest(request); >> + else if (!data->queuedControls_.empty()) >> + data->queuedControls_.pop(); >> } >> >> /** >> @@ -543,6 +604,21 @@ void PipelineHandler::doQueueRequests(Camera *camera) >> * \return 0 on success or a negative error code otherwise >> */ >> >> +/** >> + * \fn PipelineHandler::queueControlsDevice() >> + * \brief Queue controls to be applied as soon as possible >> + * \param[in] camera The camera >> + * \param[in] controls The controls to apply >> + * >> + * This function queues \a controls to \a camera so that they can be >> + * applied as soon as possible >> + * >> + * \context This function is called from the CameraManager thread. >> + * >> + * \return 0 on success or a negative error code otherwise >> + * \return -EOPNOTSUPP if fast-tracking controls is not supported >> + */ >> + >> /** >> * \brief Complete a buffer for a request >> * \param[in] request The request the buffer belongs to >> -- >> 2.47.3 >>
Hi Barnabas Thanks for the message. I think you were right about everything, but I can certainly expand a bit here and there. On Fri, 7 Aug 2026 at 18:43, Barnabás Pőcze <barnabas.pocze@ideasonboard.com> wrote: > > Hi > > 2026. 07. 23. 18:17 keltezéssel, David Plowman írta: > > Hi everyone > > > > I'd like to give this topic another prod. I talked about it a bit at > > Nice, and revisiting this patch seems like a way to start some > > discussion, though the precise implementation here is more for > > illustration. > > > > One of the principal motivations is for things like burst captures, > > where you don't want to get held up at the back of the request queue. > > > > Does this seem like a reasonable thing to do, or are there better alternatives? > > I started typing out a reply in a different thread of yours, but it got too long, > so I though it would be more effective to write a quick reply here. > > I believe there is largely agreement that some kind non-request-submission-tied control > setting mechanism is desirable. > > So I think it would be useful to examine the motivating use cases in a bit more detail. > Sorry if this was already done somewhere, I couldn't recall. > > So the way I understand it, there is a user that is cycling through a set of requests, > keeping most of them queued. This works fine, however, sometimes, maybe due to some > external event, it wants to capture a number of frames with specific settings. The > reason why applying the controls in the subsequent requests is undesirable is because > of the delay that the already queued requests introduce. Yes, agree. Burst captures are the most obvious example, perhaps. But pretty much any user event (such as changing a white balance mode) is better if it happens quickly. > > The proposed solution is to have two parallel queues: one for requests, and one for controls. > Furthermore, the two are consumed in lockstep. And there is an assumption that the control > is queue is mostly empty. This way when the user submits a control list, it can effectively > "skip" the requests in the parallel request queue, and arrive near the front of the control > queue. And thus it will be "applied to an earlier request", reducing latency. Yes, the point being that it will be applied as soon as possible. As a side note, I also quite like the API you get where to make as many control lists as you need, and you can "fire and forget" them. You don't have to manage the queue of control lists yourself by feeding the top one into a request every time one completes and you recycle it. (OK, it's not difficult, but why make an API that's deliberately annoying?) > > Is this a faithful description? If so, I have two questions: Yes, think so! > > * what if the user wants to enable/disable various streams for the "burst capture"? > do you have any thoughts about how that API would work? Indeed, this is why I raised the question the other week about being able to enable streams using a control. The kind of use case is that you're capturing only low resolution images (e.g. for preview), but then want to do a burst capture with different exposures where you also want a high resolution buffer. The obvious way to synchronise this with a queue of control lists is to turn it into a control as well. We don't have any dedicated way to handle per-stream controls, so I think we're probably left with having either an array of flags (one per stream) or a separate control per stream. But I'm definitely open to offers there! I don't see any particular problem with enabling streams in both the request and a control list, as I expect applications would be using one method or the other. The pipeline handler could simply "or" them together, or complain if they disagree, I don't think it matters very much. Though if the expectation is that control lists in a request are applied with that request, then the control is presumably equivalent to the field in the request itself? > * is it guaranteed that every control list will be applied to the results of the > corresponding request? if not, how would that work with the user enabling/disabling > various streams (during the "burst capture")? Yes, though we haven't really said what the "corresponding request" means. In my use cases, I intend to take the top control list and apply it as soon as possible. It will be applied not to the request that was at the head of the queue at that moment, but a few frames later (because it takes a few frames to apply a control list). I also expect control lists, once in the control list queue, to have a sequence number, and for completed requests to report exactly which control list (according to its sequence number) they have applied. Warning: digression follows... In the "Android" case (by which I mean the scheme where a control list in a request is applied for that request), you will find yourself looking ahead at the control lists further up the request queue, deciding what needs to be applied with the top request. It's just my opinion, but even if you don't explicitly have a queue of control lists, you do still have a kind of "virtual" control list queue here. I find the whole thing easier to think about like this. The "Android"/ "Raspberry Pi" distinction ("Raspberry Pi" being the behaviour I described previously) just becomes a question of when you consume control lists from the control list queue: "Raspberry Pi" - just consume the top control list every time. "Android" - the control list queue entries need to record their target request number, then you just consume control lists as far as topRequest->sequenceNumber + pipelineDepth. Getting all the controls to synchronise correctly is, in my experience anyway, quite tricky for pipeline handlers, so I do sometimes wonder whether there's anything libcamera can do to help. But that's perhaps a question for another time! David > > > Thanks, > Barnabás Pőcze > > > > > Thanks > > > > David > > > > On Thu, 12 Mar 2026 at 16:00, David Plowman > > <david.plowman@raspberrypi.com> wrote: > >> > >> Add `Camera::queueControls()` whose purpose is to apply controls as > >> soon as possible, without going through `Request::controls()`. > >> > >> A new virtual function `PipelineHandler::queueControlsDevice()` is > >> provided for pipeline handler to implement fast-tracked application of > >> controls. If the pipeline handler does not implement that > >> functionality, or it fails, then a fallback mechanism is used. The > >> controls will be saved for later, and they will be merged into the > >> control list of the next available request sent to the pipeline > >> handler (`Camera::Private::waitingRequests_`). > >> > >> This patch is derived directly from Barnabas's previous verion that > >> implemented the same idea but with a single ControlList, rather than > >> allowing multiple ControlLists to be queued up for consecutive > >> frames. > >> > >> Signed-off-by: David Plowman <david.plowman@raspberrypi.com> > >> --- > >> include/libcamera/camera.h | 1 + > >> include/libcamera/internal/camera.h | 1 + > >> include/libcamera/internal/pipeline_handler.h | 7 ++ > >> src/libcamera/camera.cpp | 61 +++++++++++++++ > >> src/libcamera/pipeline_handler.cpp | 76 +++++++++++++++++++ > >> 5 files changed, 146 insertions(+) > >> > >> diff --git a/include/libcamera/camera.h b/include/libcamera/camera.h > >> index b24a2974..93a484e4 100644 > >> --- a/include/libcamera/camera.h > >> +++ b/include/libcamera/camera.h > >> @@ -147,6 +147,7 @@ public: > >> > >> std::unique_ptr<Request> createRequest(uint64_t cookie = 0); > >> int queueRequest(Request *request); > >> + int queueControls(ControlList &&controls); > >> > >> int start(const ControlList *controls = nullptr); > >> int stop(); > >> diff --git a/include/libcamera/internal/camera.h b/include/libcamera/internal/camera.h > >> index 8a2e9ed5..17dda925 100644 > >> --- a/include/libcamera/internal/camera.h > >> +++ b/include/libcamera/internal/camera.h > >> @@ -38,6 +38,7 @@ public: > >> > >> std::list<Request *> queuedRequests_; > >> std::queue<Request *> waitingRequests_; > >> + std::queue<ControlList> queuedControls_; > >> ControlInfoMap controlInfo_; > >> ControlList properties_; > >> > >> diff --git a/include/libcamera/internal/pipeline_handler.h b/include/libcamera/internal/pipeline_handler.h > >> index b4f97477..c25213de 100644 > >> --- a/include/libcamera/internal/pipeline_handler.h > >> +++ b/include/libcamera/internal/pipeline_handler.h > >> @@ -57,6 +57,7 @@ public: > >> > >> void registerRequest(Request *request); > >> void queueRequest(Request *request); > >> + int queueControls(Camera *camera, ControlList controls); > >> > >> bool completeBuffer(Request *request, FrameBuffer *buffer); > >> void completeRequest(Request *request); > >> @@ -76,6 +77,12 @@ protected: > >> unsigned int useCount() const { return useCount_; } > >> > >> virtual int queueRequestDevice(Camera *camera, Request *request) = 0; > >> + > >> + virtual int queueControlsDevice([[maybe_unused]] Camera *camera, [[maybe_unused]] const ControlList &controls) > >> + { > >> + return -EOPNOTSUPP; > >> + } > >> + > >> virtual void stopDevice(Camera *camera) = 0; > >> > >> virtual bool acquireDevice(Camera *camera); > >> diff --git a/src/libcamera/camera.cpp b/src/libcamera/camera.cpp > >> index f724a1be..f0244707 100644 > >> --- a/src/libcamera/camera.cpp > >> +++ b/src/libcamera/camera.cpp > >> @@ -637,6 +637,16 @@ Camera::Private::~Private() > >> * queued requests was reached. > >> */ > >> > >> +/** > >> + * \var Camera::Private::queuedControls_ > >> + * \brief The queue of pending control lists > >> + * > >> + * This queue maintains a list of all the control lists that need to be sent > >> + * to the pipeline handler with subsequent requests. The top item in the queue > >> + * will always be sent with the next request going to > >> + * PipelineHandler::queueRequestDevice(). > >> + */ > >> + > >> /** > >> * \var Camera::Private::controlInfo_ > >> * \brief The set of controls supported by the camera > >> @@ -1378,6 +1388,57 @@ int Camera::queueRequest(Request *request) > >> return 0; > >> } > >> > >> +/** > >> + * \brief Queue controls to be applied as soon as possible > >> + * \param[in] controls The list of controls to queue > >> + * > >> + * This function tries to ensure that the controls in \a controls are applied > >> + * to the camera as soon as possible. If there are still pending controls waiting > >> + * to be applied (because of previous calls to Camera::queueControls), then > >> + * these controls will be applied as soon as possible on a frame after those. > >> + * > >> + * The exact guarantees are camera dependent, but it is guaranteed that the > >> + * controls will be applied no later than with the next \ref Request"request" > >> + * that the application \ref Camera::queueRequest() "queues" (after any requests > >> + * have been *used up" for sending previously queued controls). > >> + * > >> + * \context This function is \threadsafe. It may only be called when the camera > >> + * is in the Running state as defined in \ref camera_operation. > >> + * > >> + * \return 0 on success or a negative error code otherwise > >> + * \retval -ENODEV The camera has been disconnected from the system > >> + * \retval -EACCES The camera is not running > >> + */ > >> +int Camera::queueControls(ControlList &&controls) > >> +{ > >> + Private *const d = _d(); > >> + > >> + /* > >> + * Like requests, controls can't be queued if the camera is not running. > >> + * Controls can be applied immediately when the camera starts using the > >> + * Camera::Start method. > >> + */ > >> + > >> + int ret = d->isAccessAllowed(Private::CameraRunning); > >> + if (ret < 0) > >> + return ret; > >> + > >> + /* > >> + * We want to be able to queue empty control lists, as this gives a way of > >> + * forcing another frame with the same controls as last time, before queueing > >> + * another control list that might change them again. > >> + */ > >> + > >> + patchControlList(controls); > >> + > >> + /* > >> + * \todo Or `ConnectionTypeBlocking` to get the return value? > >> + */ > >> + d->pipe_->invokeMethod(&PipelineHandler::queueControls, ConnectionTypeQueued, this, std::move(controls)); > >> + > >> + return 0; > >> +} > >> + > >> /** > >> * \brief Start capture from camera > >> * \param[in] controls Controls to be applied before starting the Camera > >> diff --git a/src/libcamera/pipeline_handler.cpp b/src/libcamera/pipeline_handler.cpp > >> index 5c469e5b..1a87b28c 100644 > >> --- a/src/libcamera/pipeline_handler.cpp > >> +++ b/src/libcamera/pipeline_handler.cpp > >> @@ -398,6 +398,12 @@ void PipelineHandler::stop(Camera *camera) > >> ASSERT(data->queuedRequests_.empty()); > >> ASSERT(data->waitingRequests_.empty()); > >> > >> + /* > >> + * Clear out any unapplied controls. If an application wants to be > >> + * sure controls have been applied, it should wait before stopping. > >> + */ > >> + data->queuedControls_ = {}; > >> + > >> data->requestSequence_ = 0; > >> } > >> > >> @@ -477,6 +483,49 @@ void PipelineHandler::queueRequest(Request *request) > >> request->_d()->prepare(300ms); > >> } > >> > >> +/** > >> + * \brief Queue controls to apply as soon as possible > >> + * \param[in] camera The camera > >> + * \param[in] controls The controls to apply > >> + * > >> + * This function tries to queue \a controls immediately to the device by > >> + * calling queueControlsDevice(). If that fails, then a fallback mechanism > >> + * is used to ensure that \a controls will be merged into the control list > >> + * of the next available request submitted to the pipeline handler. > >> + * > >> + * \context This function is called from the CameraManager thread. > >> + */ > >> +int PipelineHandler::queueControls(Camera *camera, ControlList controls) > >> +{ > >> + Camera::Private *data = camera->_d(); > >> + int ret = queueControlsDevice(camera, controls); > >> + > >> + /* > >> + * Don't worry about later request's controls overriding the ones > >> + * sent here - the application needs to deal with that. > >> + */ > >> + > >> + if (ret == -EOPNOTSUPP) { > >> + /* > >> + * Fall back to adding the controls to the next request that enters the > >> + * pipeline handler. See PipelineHandler::doQueueRequest(). > >> + */ > >> + data->queuedControls_.push(std::move(controls)); > >> + > >> + /* Counts as "success". */ > >> + ret = 0; > >> + > >> + } else if (ret < 0) { > >> + /* > >> + * The pipeline handler is claiming to support queueControlsDevice, > >> + * but it has failed. This is an error. > >> + */ > >> + LOG(Pipeline, Debug) << "Fast tracking controls failed: " << res; > >> + } > >> + > >> + return ret; > >> +} > >> + > >> /** > >> * \brief Queue one requests to the device > >> */ > >> @@ -495,9 +544,21 @@ void PipelineHandler::doQueueRequest(Request *request) > >> return; > >> } > >> > >> + if (!data->queuedControls_.empty()) { > >> + /* > >> + * Note that `ControlList::MergePolicy::KeepExisting` is used. This is > >> + * needed to ensure that if `request` is newer than pendingControls_, > >> + * then its controls take precedence. > >> + */ > >> + request->controls().merge(data->queuedControls_.front(), > >> + ControlList::MergePolicy::KeepExisting); > >> + } > >> + > >> int ret = queueRequestDevice(camera, request); > >> if (ret) > >> cancelRequest(request); > >> + else if (!data->queuedControls_.empty()) > >> + data->queuedControls_.pop(); > >> } > >> > >> /** > >> @@ -543,6 +604,21 @@ void PipelineHandler::doQueueRequests(Camera *camera) > >> * \return 0 on success or a negative error code otherwise > >> */ > >> > >> +/** > >> + * \fn PipelineHandler::queueControlsDevice() > >> + * \brief Queue controls to be applied as soon as possible > >> + * \param[in] camera The camera > >> + * \param[in] controls The controls to apply > >> + * > >> + * This function queues \a controls to \a camera so that they can be > >> + * applied as soon as possible > >> + * > >> + * \context This function is called from the CameraManager thread. > >> + * > >> + * \return 0 on success or a negative error code otherwise > >> + * \return -EOPNOTSUPP if fast-tracking controls is not supported > >> + */ > >> + > >> /** > >> * \brief Complete a buffer for a request > >> * \param[in] request The request the buffer belongs to > >> -- > >> 2.47.3 > >> >
diff --git a/include/libcamera/camera.h b/include/libcamera/camera.h index b24a2974..93a484e4 100644 --- a/include/libcamera/camera.h +++ b/include/libcamera/camera.h @@ -147,6 +147,7 @@ public: std::unique_ptr<Request> createRequest(uint64_t cookie = 0); int queueRequest(Request *request); + int queueControls(ControlList &&controls); int start(const ControlList *controls = nullptr); int stop(); diff --git a/include/libcamera/internal/camera.h b/include/libcamera/internal/camera.h index 8a2e9ed5..17dda925 100644 --- a/include/libcamera/internal/camera.h +++ b/include/libcamera/internal/camera.h @@ -38,6 +38,7 @@ public: std::list<Request *> queuedRequests_; std::queue<Request *> waitingRequests_; + std::queue<ControlList> queuedControls_; ControlInfoMap controlInfo_; ControlList properties_; diff --git a/include/libcamera/internal/pipeline_handler.h b/include/libcamera/internal/pipeline_handler.h index b4f97477..c25213de 100644 --- a/include/libcamera/internal/pipeline_handler.h +++ b/include/libcamera/internal/pipeline_handler.h @@ -57,6 +57,7 @@ public: void registerRequest(Request *request); void queueRequest(Request *request); + int queueControls(Camera *camera, ControlList controls); bool completeBuffer(Request *request, FrameBuffer *buffer); void completeRequest(Request *request); @@ -76,6 +77,12 @@ protected: unsigned int useCount() const { return useCount_; } virtual int queueRequestDevice(Camera *camera, Request *request) = 0; + + virtual int queueControlsDevice([[maybe_unused]] Camera *camera, [[maybe_unused]] const ControlList &controls) + { + return -EOPNOTSUPP; + } + virtual void stopDevice(Camera *camera) = 0; virtual bool acquireDevice(Camera *camera); diff --git a/src/libcamera/camera.cpp b/src/libcamera/camera.cpp index f724a1be..f0244707 100644 --- a/src/libcamera/camera.cpp +++ b/src/libcamera/camera.cpp @@ -637,6 +637,16 @@ Camera::Private::~Private() * queued requests was reached. */ +/** + * \var Camera::Private::queuedControls_ + * \brief The queue of pending control lists + * + * This queue maintains a list of all the control lists that need to be sent + * to the pipeline handler with subsequent requests. The top item in the queue + * will always be sent with the next request going to + * PipelineHandler::queueRequestDevice(). + */ + /** * \var Camera::Private::controlInfo_ * \brief The set of controls supported by the camera @@ -1378,6 +1388,57 @@ int Camera::queueRequest(Request *request) return 0; } +/** + * \brief Queue controls to be applied as soon as possible + * \param[in] controls The list of controls to queue + * + * This function tries to ensure that the controls in \a controls are applied + * to the camera as soon as possible. If there are still pending controls waiting + * to be applied (because of previous calls to Camera::queueControls), then + * these controls will be applied as soon as possible on a frame after those. + * + * The exact guarantees are camera dependent, but it is guaranteed that the + * controls will be applied no later than with the next \ref Request"request" + * that the application \ref Camera::queueRequest() "queues" (after any requests + * have been *used up" for sending previously queued controls). + * + * \context This function is \threadsafe. It may only be called when the camera + * is in the Running state as defined in \ref camera_operation. + * + * \return 0 on success or a negative error code otherwise + * \retval -ENODEV The camera has been disconnected from the system + * \retval -EACCES The camera is not running + */ +int Camera::queueControls(ControlList &&controls) +{ + Private *const d = _d(); + + /* + * Like requests, controls can't be queued if the camera is not running. + * Controls can be applied immediately when the camera starts using the + * Camera::Start method. + */ + + int ret = d->isAccessAllowed(Private::CameraRunning); + if (ret < 0) + return ret; + + /* + * We want to be able to queue empty control lists, as this gives a way of + * forcing another frame with the same controls as last time, before queueing + * another control list that might change them again. + */ + + patchControlList(controls); + + /* + * \todo Or `ConnectionTypeBlocking` to get the return value? + */ + d->pipe_->invokeMethod(&PipelineHandler::queueControls, ConnectionTypeQueued, this, std::move(controls)); + + return 0; +} + /** * \brief Start capture from camera * \param[in] controls Controls to be applied before starting the Camera diff --git a/src/libcamera/pipeline_handler.cpp b/src/libcamera/pipeline_handler.cpp index 5c469e5b..1a87b28c 100644 --- a/src/libcamera/pipeline_handler.cpp +++ b/src/libcamera/pipeline_handler.cpp @@ -398,6 +398,12 @@ void PipelineHandler::stop(Camera *camera) ASSERT(data->queuedRequests_.empty()); ASSERT(data->waitingRequests_.empty()); + /* + * Clear out any unapplied controls. If an application wants to be + * sure controls have been applied, it should wait before stopping. + */ + data->queuedControls_ = {}; + data->requestSequence_ = 0; } @@ -477,6 +483,49 @@ void PipelineHandler::queueRequest(Request *request) request->_d()->prepare(300ms); } +/** + * \brief Queue controls to apply as soon as possible + * \param[in] camera The camera + * \param[in] controls The controls to apply + * + * This function tries to queue \a controls immediately to the device by + * calling queueControlsDevice(). If that fails, then a fallback mechanism + * is used to ensure that \a controls will be merged into the control list + * of the next available request submitted to the pipeline handler. + * + * \context This function is called from the CameraManager thread. + */ +int PipelineHandler::queueControls(Camera *camera, ControlList controls) +{ + Camera::Private *data = camera->_d(); + int ret = queueControlsDevice(camera, controls); + + /* + * Don't worry about later request's controls overriding the ones + * sent here - the application needs to deal with that. + */ + + if (ret == -EOPNOTSUPP) { + /* + * Fall back to adding the controls to the next request that enters the + * pipeline handler. See PipelineHandler::doQueueRequest(). + */ + data->queuedControls_.push(std::move(controls)); + + /* Counts as "success". */ + ret = 0; + + } else if (ret < 0) { + /* + * The pipeline handler is claiming to support queueControlsDevice, + * but it has failed. This is an error. + */ + LOG(Pipeline, Debug) << "Fast tracking controls failed: " << res; + } + + return ret; +} + /** * \brief Queue one requests to the device */ @@ -495,9 +544,21 @@ void PipelineHandler::doQueueRequest(Request *request) return; } + if (!data->queuedControls_.empty()) { + /* + * Note that `ControlList::MergePolicy::KeepExisting` is used. This is + * needed to ensure that if `request` is newer than pendingControls_, + * then its controls take precedence. + */ + request->controls().merge(data->queuedControls_.front(), + ControlList::MergePolicy::KeepExisting); + } + int ret = queueRequestDevice(camera, request); if (ret) cancelRequest(request); + else if (!data->queuedControls_.empty()) + data->queuedControls_.pop(); } /** @@ -543,6 +604,21 @@ void PipelineHandler::doQueueRequests(Camera *camera) * \return 0 on success or a negative error code otherwise */ +/** + * \fn PipelineHandler::queueControlsDevice() + * \brief Queue controls to be applied as soon as possible + * \param[in] camera The camera + * \param[in] controls The controls to apply + * + * This function queues \a controls to \a camera so that they can be + * applied as soon as possible + * + * \context This function is called from the CameraManager thread. + * + * \return 0 on success or a negative error code otherwise + * \return -EOPNOTSUPP if fast-tracking controls is not supported + */ + /** * \brief Complete a buffer for a request * \param[in] request The request the buffer belongs to
Add `Camera::queueControls()` whose purpose is to apply controls as soon as possible, without going through `Request::controls()`. A new virtual function `PipelineHandler::queueControlsDevice()` is provided for pipeline handler to implement fast-tracked application of controls. If the pipeline handler does not implement that functionality, or it fails, then a fallback mechanism is used. The controls will be saved for later, and they will be merged into the control list of the next available request sent to the pipeline handler (`Camera::Private::waitingRequests_`). This patch is derived directly from Barnabas's previous verion that implemented the same idea but with a single ControlList, rather than allowing multiple ControlLists to be queued up for consecutive frames. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> --- include/libcamera/camera.h | 1 + include/libcamera/internal/camera.h | 1 + include/libcamera/internal/pipeline_handler.h | 7 ++ src/libcamera/camera.cpp | 61 +++++++++++++++ src/libcamera/pipeline_handler.cpp | 76 +++++++++++++++++++ 5 files changed, 146 insertions(+)