From patchwork Sat Aug 31 21:02:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Niklas_S=C3=B6derlund?= X-Patchwork-Id: 1903 X-Patchwork-Delegate: niklas.soderlund@ragnatech.se Return-Path: Received: from vsp-unauthed02.binero.net (vsp-unauthed02.binero.net [195.74.38.227]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 9E30460C18 for ; Sat, 31 Aug 2019 23:02:59 +0200 (CEST) X-Halon-ID: b4de6b9b-cc32-11e9-903a-005056917f90 Authorized-sender: niklas@soderlund.pp.se Received: from bismarck.berto.se (unknown [84.172.84.18]) by bin-vsp-out-02.atm.binero.net (Halon) with ESMTPA id b4de6b9b-cc32-11e9-903a-005056917f90; Sat, 31 Aug 2019 23:02:57 +0200 (CEST) From: =?utf-8?q?Niklas_S=C3=B6derlund?= To: libcamera-devel@lists.libcamera.org Date: Sat, 31 Aug 2019 23:02:15 +0200 Message-Id: <20190831210220.29819-2-niklas.soderlund@ragnatech.se> X-Mailer: git-send-email 2.22.1 In-Reply-To: <20190831210220.29819-1-niklas.soderlund@ragnatech.se> References: <20190831210220.29819-1-niklas.soderlund@ragnatech.se> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 1/6] libcamera: request: Add IPAMetaData X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Aug 2019 21:03:00 -0000 Add a new structure to hold meta data coming out of the IPA. The structure will grow over time but for now only add information about the auto exposure state as it can be directly used by the rkisp1 IPA, which is capable of controlling exposure. Signed-off-by: Niklas Söderlund --- include/libcamera/request.h | 13 +++++++++++ src/libcamera/request.cpp | 44 +++++++++++++++++++++++++++++++++++-- 2 files changed, 55 insertions(+), 2 deletions(-) diff --git a/include/libcamera/request.h b/include/libcamera/request.h index f5de5257bba3f2bb..6f75a59fb69a43d7 100644 --- a/include/libcamera/request.h +++ b/include/libcamera/request.h @@ -21,6 +21,17 @@ class Buffer; class Camera; class Stream; +enum AeState { + Inactive, + Searching, + Converged, +}; + +struct IPAMetaData { + AeState aeState; + bool ready; +}; + class RequestData { }; @@ -44,6 +55,7 @@ public: const std::map &buffers() const { return bufferMap_; } int addBuffer(std::unique_ptr buffer); Buffer *findBuffer(Stream *stream) const; + const IPAMetaData &metaData() const { return metaData_; }; uint64_t cookie() const { return cookie_; } Status status() const { return status_; } @@ -65,6 +77,7 @@ private: ControlList controls_; std::map bufferMap_; std::unordered_set pending_; + IPAMetaData metaData_; const uint64_t cookie_; Status status_; diff --git a/src/libcamera/request.cpp b/src/libcamera/request.cpp index bdb4f908fb664ccf..f40cf5cac1f029a9 100644 --- a/src/libcamera/request.cpp +++ b/src/libcamera/request.cpp @@ -24,6 +24,40 @@ namespace libcamera { LOG_DEFINE_CATEGORY(Request) +/** + * \enum AeState + * State of Auto Exposure algorithm + * \var AeState::Inactive + * AE not running + * \var AeState::Searching + * AE is not converged to a good value and is adjusting exposure parameters. + * \var AeState::Converged + * AE has found good exposure values for the current scene. + */ + +/** + * \struct IPAMetaData + * \brief Meta data describing the state of the IPA + * + * Container for IPA meta data. The intended creator of this object is an IPA + * and the intended consumer is applications. Applications access the object + * thru the Request object that corresponds to the specific capture event + * that generated the meta data. + */ + +/** + * \var IPAMetaData::aeState + * \brief Holds the state of the Auto Exposure algorithm + */ + +/** + * \var IPAMetaData::ready + * \brief Flag to indicate the pipeline have validated the meta data + * + * The meta data should not be returned to the application by the specific + * pipeline handler implementation before this flag is set to true. + */ + /** * \enum Request::Status * Request completion status @@ -55,8 +89,8 @@ LOG_DEFINE_CATEGORY(Request) * */ Request::Request(Camera *camera, uint64_t cookie) - : data(nullptr), camera_(camera), controls_(camera), cookie_(cookie), - status_(RequestPending), cancelled_(false) + : data(nullptr), camera_(camera), controls_(camera), metaData_({}), + cookie_(cookie), status_(RequestPending), cancelled_(false) { } @@ -156,6 +190,12 @@ Buffer *Request::findBuffer(Stream *stream) const return it->second; } +/** + * \fn Request::metaData() + * \brief Retrieve the request's meta data + * \return The meta data associated with the request + */ + /** * \fn Request::cookie() * \brief Retrieve the cookie set when the request was created From patchwork Sat Aug 31 21:02:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Niklas_S=C3=B6derlund?= X-Patchwork-Id: 1904 X-Patchwork-Delegate: niklas.soderlund@ragnatech.se Return-Path: Received: from bin-mail-out-06.binero.net (bin-mail-out-06.binero.net [195.74.38.229]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 2FE2560BB2 for ; Sat, 31 Aug 2019 23:03:00 +0200 (CEST) X-Halon-ID: b55c4361-cc32-11e9-903a-005056917f90 Authorized-sender: niklas@soderlund.pp.se Received: from bismarck.berto.se (unknown [84.172.84.18]) by bin-vsp-out-02.atm.binero.net (Halon) with ESMTPA id b55c4361-cc32-11e9-903a-005056917f90; Sat, 31 Aug 2019 23:02:58 +0200 (CEST) From: =?utf-8?q?Niklas_S=C3=B6derlund?= To: libcamera-devel@lists.libcamera.org Date: Sat, 31 Aug 2019 23:02:16 +0200 Message-Id: <20190831210220.29819-3-niklas.soderlund@ragnatech.se> X-Mailer: git-send-email 2.22.1 In-Reply-To: <20190831210220.29819-1-niklas.soderlund@ragnatech.se> References: <20190831210220.29819-1-niklas.soderlund@ragnatech.se> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 2/6] libcamera: ipa: Add signal to transferee meta data X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Aug 2019 21:03:00 -0000 Add a signal which will be emitted once the IPA have processed the statistics and translated it into meta data libcamera can understand. Signed-off-by: Niklas Söderlund --- include/libcamera/ipa/ipa_interface.h | 2 ++ src/libcamera/ipa_interface.cpp | 10 ++++++++++ 2 files changed, 12 insertions(+) diff --git a/include/libcamera/ipa/ipa_interface.h b/include/libcamera/ipa/ipa_interface.h index 38e16ff37214f7b9..eace4d3ea97943d0 100644 --- a/include/libcamera/ipa/ipa_interface.h +++ b/include/libcamera/ipa/ipa_interface.h @@ -16,6 +16,7 @@ namespace libcamera { class Buffer; class Request; +struct IPAMetaData; class IPAInterface { @@ -30,6 +31,7 @@ public: Signal updateSensor; Signal queueRequest; + Signal metaDataReady; }; } /* namespace libcamera */ diff --git a/src/libcamera/ipa_interface.cpp b/src/libcamera/ipa_interface.cpp index be9eab3cda32379d..3222ae09e5750030 100644 --- a/src/libcamera/ipa_interface.cpp +++ b/src/libcamera/ipa_interface.cpp @@ -92,4 +92,14 @@ namespace libcamera { * controls. */ +/** + * \var IPAInterface::metaDataReady + * \brief Signal emitted when the IPA is done processing statistics + * + * This signal is emitted when the IPA have finished processing the statistics + * buffer and have created an IPAMetaData object which are ready to be consumed + * by the pipeline handler. The request cookie and the meta data is passed as + * parameters. + */ + } /* namespace libcamera */ From patchwork Sat Aug 31 21:02:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Niklas_S=C3=B6derlund?= X-Patchwork-Id: 1905 X-Patchwork-Delegate: niklas.soderlund@ragnatech.se Return-Path: Received: from bin-mail-out-05.binero.net (bin-mail-out-05.binero.net [195.74.38.228]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id D0C5C60C18 for ; Sat, 31 Aug 2019 23:03:00 +0200 (CEST) X-Halon-ID: b60ef0ae-cc32-11e9-903a-005056917f90 Authorized-sender: niklas@soderlund.pp.se Received: from bismarck.berto.se (unknown [84.172.84.18]) by bin-vsp-out-02.atm.binero.net (Halon) with ESMTPA id b60ef0ae-cc32-11e9-903a-005056917f90; Sat, 31 Aug 2019 23:02:59 +0200 (CEST) From: =?utf-8?q?Niklas_S=C3=B6derlund?= To: libcamera-devel@lists.libcamera.org Date: Sat, 31 Aug 2019 23:02:17 +0200 Message-Id: <20190831210220.29819-4-niklas.soderlund@ragnatech.se> X-Mailer: git-send-email 2.22.1 In-Reply-To: <20190831210220.29819-1-niklas.soderlund@ragnatech.se> References: <20190831210220.29819-1-niklas.soderlund@ragnatech.se> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 3/6] libcamera: pipeline: Add helper to process meta data coming from IPA X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Aug 2019 21:03:01 -0000 Add a helper to process meta data coming out of an IPA and associating it with a request. The helper don't do much yet, it's expected that more code will be needed in the future to deal with versioning of the meta data structure coming from the IPA and the structure provided to applications. Signed-off-by: Niklas Söderlund --- src/libcamera/include/pipeline_handler.h | 2 ++ src/libcamera/pipeline_handler.cpp | 16 ++++++++++++++++ 2 files changed, 18 insertions(+) diff --git a/src/libcamera/include/pipeline_handler.h b/src/libcamera/include/pipeline_handler.h index ca40e7ef2b85f372..3c08ecb0260675e0 100644 --- a/src/libcamera/include/pipeline_handler.h +++ b/src/libcamera/include/pipeline_handler.h @@ -102,6 +102,8 @@ protected: void prepareInternalBuffer(Buffer *buffer, Request *request, BufferMemory *mem); + void processMetaData(Request *request, const IPAMetaData &metaData); + CameraManager *manager_; private: diff --git a/src/libcamera/pipeline_handler.cpp b/src/libcamera/pipeline_handler.cpp index 613751619a398968..04293921d4d81d98 100644 --- a/src/libcamera/pipeline_handler.cpp +++ b/src/libcamera/pipeline_handler.cpp @@ -523,6 +523,22 @@ void PipelineHandler::prepareInternalBuffer(Buffer *buffer, Request *request, buffer->mem_ = mem; } +/** + * \brief Helper to process meta data from the IPA + * \param[in] request The request to associate the \a metaData with + * \param[in] metaData The meta data to process + * + * This function is a helper for pipline handler implementations to process + * meta data retrived from an IPA. It is mandatory to call this function with + * any meta data returned from the IPA before it's passed to the application. + */ +void PipelineHandler::processMetaData(Request *request, + const IPAMetaData &metaData) +{ + request->metaData_ = metaData; + request->metaData_.ready = true; +} + /** * \brief Slot for the MediaDevice disconnected signal */ From patchwork Sat Aug 31 21:02:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Niklas_S=C3=B6derlund?= X-Patchwork-Id: 1907 X-Patchwork-Delegate: niklas.soderlund@ragnatech.se Return-Path: Received: from bin-mail-out-06.binero.net (bin-mail-out-06.binero.net [195.74.38.229]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 6A88760BB2 for ; Sat, 31 Aug 2019 23:03:02 +0200 (CEST) X-Halon-ID: b67a5b4d-cc32-11e9-903a-005056917f90 Authorized-sender: niklas@soderlund.pp.se Received: from bismarck.berto.se (unknown [84.172.84.18]) by bin-vsp-out-02.atm.binero.net (Halon) with ESMTPA id b67a5b4d-cc32-11e9-903a-005056917f90; Sat, 31 Aug 2019 23:02:59 +0200 (CEST) From: =?utf-8?q?Niklas_S=C3=B6derlund?= To: libcamera-devel@lists.libcamera.org Date: Sat, 31 Aug 2019 23:02:18 +0200 Message-Id: <20190831210220.29819-5-niklas.soderlund@ragnatech.se> X-Mailer: git-send-email 2.22.1 In-Reply-To: <20190831210220.29819-1-niklas.soderlund@ragnatech.se> References: <20190831210220.29819-1-niklas.soderlund@ragnatech.se> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 4/6] libcamera: ipa: rkisp1: Fill in meta data about auto exposure X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Aug 2019 21:03:03 -0000 Fill in the meta data parsed from the statistics buffer and use the metaDataReady signal to notify the pipeline handler that the data is ready. The method to judge if auto exposure is converged or not is just as simple as the algorithm controlling the exposure time. If we are +/- 5% from our target value we are converged. Signed-off-by: Niklas Söderlund --- src/ipa/ipa_rkisp1.cpp | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-) diff --git a/src/ipa/ipa_rkisp1.cpp b/src/ipa/ipa_rkisp1.cpp index 063b075d8358c30d..cc926867d6fde478 100644 --- a/src/ipa/ipa_rkisp1.cpp +++ b/src/ipa/ipa_rkisp1.cpp @@ -6,6 +6,7 @@ */ #include +#include #include #include @@ -112,8 +113,9 @@ void IPARkISP1::updateStatistics(const void *cookie, Buffer &statistics) const rkisp1_stat_buffer *stats = static_cast(statistics.mem()->planes()[0].mem()); const cifisp_stat *params = &stats->params; + IPAMetaData metaData = {}; - if ((stats->meas_type & CIFISP_STAT_AUTOEXP) && (statFrame_ % 2 == 0)) { + if (stats->meas_type & CIFISP_STAT_AUTOEXP) { const cifisp_ae_stat *ae = ¶ms->ae; const unsigned int target = 60; @@ -129,18 +131,28 @@ void IPARkISP1::updateStatistics(const void *cookie, Buffer &statistics) value /= num; double factor = (double)target / value; - double tmp; - tmp = factor * exposure_ * gain_ / minGain_; - exposure_ = utils::clamp((uint64_t)tmp, minExposure_, maxExposure_); + if (statFrame_ % 3 == 0) { + double tmp; - tmp = tmp / exposure_ * minGain_; - gain_ = utils::clamp((uint64_t)tmp, minGain_, maxGain_); + tmp = factor * exposure_ * gain_ / minGain_; + exposure_ = utils::clamp((uint64_t)tmp, minExposure_, maxExposure_); - setControls(); + tmp = tmp / exposure_ * minGain_; + gain_ = utils::clamp((uint64_t)tmp, minGain_, maxGain_); + + setControls(); + } + + metaData.aeState = fabs(factor - 1.0f) < 0.05f ? + AeState::Converged : AeState::Searching; + } else { + metaData.aeState = AeState::Inactive; } statFrame_++; + + metaDataReady.emit(cookie, metaData); } /* From patchwork Sat Aug 31 21:02:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Niklas_S=C3=B6derlund?= X-Patchwork-Id: 1906 X-Patchwork-Delegate: niklas.soderlund@ragnatech.se Return-Path: Received: from vsp-unauthed02.binero.net (vsp-unauthed02.binero.net [195.74.38.227]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 6B4E160E2A for ; Sat, 31 Aug 2019 23:03:02 +0200 (CEST) X-Halon-ID: b6e87e3a-cc32-11e9-903a-005056917f90 Authorized-sender: niklas@soderlund.pp.se Received: from bismarck.berto.se (unknown [84.172.84.18]) by bin-vsp-out-02.atm.binero.net (Halon) with ESMTPA id b6e87e3a-cc32-11e9-903a-005056917f90; Sat, 31 Aug 2019 23:03:00 +0200 (CEST) From: =?utf-8?q?Niklas_S=C3=B6derlund?= To: libcamera-devel@lists.libcamera.org Date: Sat, 31 Aug 2019 23:02:19 +0200 Message-Id: <20190831210220.29819-6-niklas.soderlund@ragnatech.se> X-Mailer: git-send-email 2.22.1 In-Reply-To: <20190831210220.29819-1-niklas.soderlund@ragnatech.se> References: <20190831210220.29819-1-niklas.soderlund@ragnatech.se> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 5/6] libcamera: pipeline: rkisp1: Add plumbing for IPA meta data X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Aug 2019 21:03:03 -0000 Connect the pipeline handler to the IPA meta data signal. Add a check for that IPA meta data is available to the conditions for completing a request to the application. Signed-off-by: Niklas Söderlund --- src/libcamera/pipeline/rkisp1/rkisp1.cpp | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp index ab7c595f1e3bd39f..501edc20bb43514f 100644 --- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp +++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp @@ -51,6 +51,7 @@ public: private: void updateSensor(V4L2ControlList controls); void queueRequestHardware(const void *cookie); + void metaDataReady(const void *cookie, IPAMetaData metaData); }; class RkISP1RequestData : public RequestData @@ -143,6 +144,8 @@ int RkISP1CameraData::initIPA() &RkISP1CameraData::updateSensor); ipa_->queueRequest.connect(this, &RkISP1CameraData::queueRequestHardware); + ipa_->metaDataReady.connect(this, + &RkISP1CameraData::metaDataReady); return 0; } @@ -175,6 +178,17 @@ void RkISP1CameraData::queueRequestHardware(const void *cookie) LOG(RkISP1, Error) << "Failed to queue video"; } +void RkISP1CameraData::metaDataReady(const void *cookie, IPAMetaData metaData) +{ + Request *request = reinterpret_cast(const_cast(cookie)); + PipelineHandlerRkISP1 *pipe = + static_cast(pipe_); + + pipe->processMetaData(request, metaData); + + pipe->tryCompleteRequest(request); +} + RkISP1CameraConfiguration::RkISP1CameraConfiguration(Camera *camera, RkISP1CameraData *data) : CameraConfiguration() @@ -709,6 +723,9 @@ void PipelineHandlerRkISP1::tryCompleteRequest(Request *request) if (request->hasPendingBuffers()) return; + if (!request->metaData().ready) + return; + delete reqData; request->data = nullptr; @@ -734,8 +751,6 @@ void PipelineHandlerRkISP1::statReady(Buffer *buffer) data->ipa_->updateStatistics(request, *buffer); - /* TODO: Fetch libcamera status controls from IPA */ - reqData->stat = nullptr; statBuffers_.push(buffer); From patchwork Sat Aug 31 21:02:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Niklas_S=C3=B6derlund?= X-Patchwork-Id: 1908 X-Patchwork-Delegate: niklas.soderlund@ragnatech.se Return-Path: Received: from vsp-unauthed02.binero.net (vsp-unauthed02.binero.net [195.74.38.227]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id EC7A960C1E for ; Sat, 31 Aug 2019 23:03:02 +0200 (CEST) X-Halon-ID: b75f2b98-cc32-11e9-903a-005056917f90 Authorized-sender: niklas@soderlund.pp.se Received: from bismarck.berto.se (unknown [84.172.84.18]) by bin-vsp-out-02.atm.binero.net (Halon) with ESMTPA id b75f2b98-cc32-11e9-903a-005056917f90; Sat, 31 Aug 2019 23:03:01 +0200 (CEST) From: =?utf-8?q?Niklas_S=C3=B6derlund?= To: libcamera-devel@lists.libcamera.org Date: Sat, 31 Aug 2019 23:02:20 +0200 Message-Id: <20190831210220.29819-7-niklas.soderlund@ragnatech.se> X-Mailer: git-send-email 2.22.1 In-Reply-To: <20190831210220.29819-1-niklas.soderlund@ragnatech.se> References: <20190831210220.29819-1-niklas.soderlund@ragnatech.se> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 6/6] qcam: Print auto exposure state for each completed request X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Aug 2019 21:03:03 -0000 Print the auto exposure state to stdout when a requests complete. Signed-off-by: Niklas Söderlund --- src/qcam/main_window.cpp | 1 + 1 file changed, 1 insertion(+) diff --git a/src/qcam/main_window.cpp b/src/qcam/main_window.cpp index b39cc8ad942dacaa..343123722fd3350c 100644 --- a/src/qcam/main_window.cpp +++ b/src/qcam/main_window.cpp @@ -257,6 +257,7 @@ void MainWindow::requestComplete(Request *request, << " bytesused: " << buffer->bytesused() << " timestamp: " << buffer->timestamp() << " fps: " << std::fixed << std::setprecision(2) << fps + << " ae: " << request->metaData().aeState << std::endl; display(buffer);