From patchwork Mon Jun 15 10:50:02 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Kieran Bingham X-Patchwork-Id: 4037 Return-Path: Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id A3BC5603D8 for ; Mon, 15 Jun 2020 12:50:06 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=pass (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="tHJmgKI6"; dkim-atps=neutral Received: from Q.local (cpc89242-aztw30-2-0-cust488.18-1.cable.virginm.net [86.31.129.233]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 039AFF9; Mon, 15 Jun 2020 12:50:05 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1592218206; bh=eR1+8NWUHN+ZjLcstwrm8TSRB/2tx1K9Dla9V948WEQ=; h=From:To:Cc:Subject:Date:From; b=tHJmgKI6cf9TjEZb/eW9zXAZaU6q24UM0fYJMOg0Hw929hPD4nORKoxqxlRkmMVyx Tr5Ec+5E8LcnyiPpyOVcD3w1gwC51YOy3Gt/bu1wwVITCFBL/Br2MffYfUCjootLrA rC2pfhIc6VlNGwH+9ULjnKnahyhquAHJj2rZ3kL8= From: Kieran Bingham To: libcamera devel Cc: Chris Ward Date: Mon, 15 Jun 2020 11:50:02 +0100 Message-Id: <20200615105002.555588-1-kieran.bingham@ideasonboard.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH-Resend] Add getting started guide for application developers X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Jun 2020 10:50:07 -0000 From: Chris Chinchilla Reviewed-by: Paul Elder --- [Kieran:] Resending this inline to ease review on the libcamera mailing-list. .../guides/libcamera-application-author.rst | 472 ++++++++++++++++++ 1 file changed, 472 insertions(+) create mode 100644 Documentation/guides/libcamera-application-author.rst diff --git a/Documentation/guides/libcamera-application-author.rst b/Documentation/guides/libcamera-application-author.rst new file mode 100644 index 000000000000..c5f723820004 --- /dev/null +++ b/Documentation/guides/libcamera-application-author.rst @@ -0,0 +1,472 @@ +Supporting libcamera in your application +======================================== + +This tutorial shows you how to create an application that uses libcamera +to connect to a camera on a system, capture frames from it for 3 +seconds, and write metadata about the frames to standard out. + +.. TODO: How much of the example code runs before camera start etc? + +Create a pointer to the camera +------------------------------ + +Before the ``int main()`` function, create a global shared pointer +variable for the camera: + +.. code:: cpp + + std::shared_ptr camera; + + int main() + { + // Code to follow + } + +Camera Manager +-------------- + +Every libcamera-based application needs an instance of a +`CameraManager `_ +that runs for the life of the application. When you start the Camera +Manager, it finds all the cameras available to the current system. +Behind the scenes, the libcamera Pipeline Handler abstracts and manages +the complex pipelines that kernel drivers expose through the `Linux +Media +Controller `__ +and `V4L2 `__ APIs, meaning that an +application doesn’t need to handle device or driver specifics. + +To create and start a new Camera Manager, create a new pointer variable +to the instance, and then start it: + +.. code:: cpp + + CameraManager *cm = new CameraManager(); + cm->start(); + +When you build the application, the Camera Manager identifies all +supported devices and creates cameras the application can interact with. + +The code below identifies all available cameras, and for this example, +writes them to standard output: + +.. code:: cpp + + for (auto const &camera : cm->cameras()) + std::cout << camera->name() << std::endl; + +For example, the output on Ubuntu running in a VM on macOS is +``FaceTime HD Camera (Built-in):``, and for x y, etc. + +.. TODO: Better examples + +Create and acquire a camera +--------------------------- + +What libcamera considers a camera +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The libcamera library supports fixed and hot-pluggable cameras, +including cameras, plugged and unplugged after initializing the library. +The libcamera library supports point-and-shoot still image and video +capture, either controlled directly by the CPU or exposed through an +internal USB bus as a UVC device designed for video conferencing usage. +The libcamera library considers any device that includes independent +camera sensors, such as front and back sensors as multiple different +camera devices. + +Once you know what camera you want to use, your application needs to +acquire a lock to it so no other application can use it. + +This example application uses a single camera that the Camera Manager +reports as available to applications. + +The code below creates the name of the first available camera as a +convenience variable, fetches that camera, and acquires the device for +exclusive access: + +.. code:: cpp + + std::string cameraName = cm->cameras()[0]->name(); + camera = cm->get(cameraName); + camera->acquire(); + +Configure the camera +-------------------- + +Before the application can do anything with the camera, you need to know +what it’s capabilities are. These capabilities include scalars, +resolutions, supported formats and converters. The libcamera library +uses ``StreamRole``\ s to define four predefined ways an application +intends to use a camera (`You can read the full list in the API +documentation `__). + +To find out if how your application wants to use the camera is possible, +generate a new configuration using a vector of ``StreamRole``\ s, and +send that vector to the camera. To do this, create a new configuration +variable and use the ``generateConfiguration`` function to produce a +``CameraConfiguration`` for it. If the camera can handle the +configuration it returns a full ``CameraConfiguration``, and if it +can’t, a null pointer. + +.. code:: cpp + + std::unique_ptr config = camera->generateConfiguration( { StreamRole::Viewfinder } ); + +A ``CameraConfiguration`` has a ``StreamConfiguration`` instance for +each ``StreamRole`` the application requested, and that the camera can +support. Each of these has a default size and format that the camera +assigned, depending on the ``StreamRole`` requested. + +The code below creates a new ``StreamConfiguration`` variable and +populates it with the value of the first (and only) ``StreamRole`` in +the camera configuration. It then outputs the value to standard out. + +.. code:: cpp + + StreamConfiguration &streamConfig = config->at(0); + std::cout << "Default viewfinder configuration is: " << streamConfig.toString() << std::endl; + +Change and validate the configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Once you have a ``StreamConfiguration``, your application can make +changes to the parameters it contains, for example, to change the width +and height you could use the following code: + +.. code:: cpp + + streamConfig.size.width = 640; + streamConfig.size.height = 480; + +If your application makes changes to any parameters, validate them +before applying them to the camera by using the ``validate`` function. +If the new value is invalid, the validation process adjusts the +parameter to what it considers a valid value. An application should +check that the adjusted configuration is something you expect (you can +use the +`Status `_ +method to check if the Pipeline Handler adjusted the configuration). + +For example, above you set the width and height to 640x480, but if the +camera cannot produce an image that large, it might return the +configuration with a new size of 320x240 and a status of ``Adjusted``. + +For this example application, the code below prints the adjusted values +to standard out. + +.. code:: cpp + + config->validate(); + std::cout << "Validated viewfinder configuration is: " << streamConfig.toString() << std::endl; + +With a validated ``CameraConfiguration``, send it to the camera to +confirm the new configuration: + +.. code:: cpp + + camera->configure(config.get()); + +If you don’t first validate the configuration before calling +``configure``, there’s a chance that calling the function fails. + +Allocate FrameBuffers +--------------------- + +The libcamera library consumes buffers provided by applications as +``FrameBuffer`` instances, which makes libcamera a consumer of buffers +exported by other devices (such as displays or video encoders), or +allocated from an external allocator (such as ION on Android). + +The libcamera library uses ``FrameBuffer`` instances to buffer frames of +data from memory, but first, your application should reserve enough +memory for the ``FrameBuffer``\ s your streams need based on the sizes +and formats you configured. + +In some situations, applications do not have any means to allocate or +get hold of suitable buffers, for instance, when no other device is +involved, or on Linux platforms that lack a centralized allocator. The +``FrameBufferAllocator`` class provides a buffer allocator that you can +use in these situations. + +An application doesn’t have to use the default ``FrameBufferAllocator`` +that libcamera provides, and can instead allocate memory manually, and +pass the buffers in ``Request``\ s (read more about ``Request``\ s in +`the frame capture section <#frame-capture>`__ of this guide). The +example in this guide covers using the ``FrameBufferAllocator`` that +libcamera provides. + +Using the libcamera ``FrameBufferAllocator`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +As the camera manager knows what configuration is available, it can +allocate all the resources for you with a single method, and de-allocate +with another. + +Applications create a ``FrameBufferAllocator`` for a Camera, and use it +to allocate buffers for streams of a ``CameraConfiguration`` with the +``allocate()`` function. + +.. code:: cpp + + for (StreamConfiguration &cfg : *config) { + int ret = allocator->allocate(cfg.stream()); + if (ret < 0) { + std::cerr << "Can't allocate buffers" << std::endl; + return -ENOMEM; + } + + unsigned int allocated = allocator->buffers(cfg.stream()).size(); + std::cout << "Allocated " << allocated << " buffers for stream" << std::endl; + } + +Frame Capture +~~~~~~~~~~~~~ + +The libcamera library follows a familiar streaming request model for +data (frames in this case). For each frame a camera captures, your +application must queue a request for it to the camera. + +In the case of libcamera, a ‘Request’ is at least one Stream (one source +from a Camera), with a FrameBuffer full of image data. + +First, create an instance of a ``Stream`` from the ``StreamConfig``, +assign a vector of ``FrameBuffer``\ s to the allocation created above, +and create a vector of the requests the application will make. + +.. code:: cpp + + Stream *stream = streamConfig.stream(); + const std::vector> &buffers = allocator->buffers(stream); + std::vector requests; + +Create ``Request``\ s for the size of the ``FrameBuffer`` by using the +``createRequest`` function and adding all the requests created to a +vector. For each request add a buffer to it with the ``addBuffer`` +function, passing the stream the buffer belongs to, and the ``FrameBuffer``. + +.. code:: cpp + + for (unsigned int i = 0; i < buffers.size(); ++i) { + Request *request = camera->createRequest(); + if (!request) + { + std::cerr << "Can't create request" << std::endl; + return -ENOMEM; + } + + const std::unique_ptr &buffer = buffers[i]; + int ret = request->addBuffer(stream, buffer.get()); + if (ret < 0) + { + std::cerr << "Can't set buffer for request" + << std::endl; + return ret; + } + + requests.push_back(request); + + /* + * todo: Set controls + * + * ControlList &Request::controls(); + * controls.set(controls::Brightness, 255); + */ + } + +.. TODO: Controls +.. TODO: A request can also have controls or parameters that you can apply to the image. --> + +Event handling and callbacks +---------------------------- + +The libcamera library uses the concept of signals and slots (`similar to +Qt `__) to connect events +with callbacks to handle those events. + +Signals +~~~~~~~ + +Signals are emitted when the buffer has been completed (image data +written into), and because a Request can contain multiple buffers - the +Request completed signal is emitted when all buffers within the request +are completed. + +A camera class instance emits a completed request signal to report when +all the buffers in a request are complete with image data written to +them. To receive these signals, connect a slot function to the signal +you are interested in. For this example application, that’s when the +camera completes a request. + +.. code:: cpp + + camera->requestCompleted.connect(requestComplete); + +Slots +~~~~~ + +Every time the camera request completes, it emits a signal, and the +connected slot invoked, passing the Request as a parameter. + +For this example application, the ``requestComplete`` slot outputs +information about the ``FrameBuffer`` to standard out, but the callback is +typically where your application accesses the image data from the camera +and does something with it. + +Signals operate in the libcamera ``CameraManager`` thread context, so it +is important not to block the thread for a long time, as this blocks +internal processing of the camera pipelines, and can affect realtime +performances, leading to skipped frames etc. + +First, create the function that matches the slot: + +.. code:: cpp + + static void requestComplete(Request *request) + { + // Code to follow + } + +The signal/slot flow is the only way to pass requests and buffers from +libcamera back to the application. There are times when a request can +emit a ``requestComplete`` signal, but this request is actually +cancelled, for example by application shutdown. To avoid an application +processing image data that doesn’t exist, it’s worth checking that the +request is still in the state you expect (You can find `a full list of +the completion statuses in the +documentation `__). + +.. code:: cpp + + if (request->status() == Request::RequestCancelled) return; + +When the request completes, you can access the buffers from the request +using the ``buffers()`` function which returns a map of each buffer, and +the stream it is associated with. + +.. code:: cpp + + const std::map &buffers = request->buffers(); + +Iterating through the map allows you to inspect each buffer from each +stream completed in this request, and access the metadata for each frame +the camera captured. The buffer metadata contains information such as +capture status, a timestamp and the bytes used. + +.. code:: cpp + + for (auto bufferPair : buffers) { + FrameBuffer *buffer = bufferPair.second; + const FrameMetadata &metadata = buffer->metadata(); + } + +The buffer describes the image data, but a buffer can consist of more +than one image plane that in memory hold the image data in the Frames. +For example, the Y, U, and V components of a YUV-encoded image are +described by a plane. + +For this example application, still inside the ``for`` loop from above, +print the Frame sequence number and details of the planes. + +.. code:: cpp + + std::cout << " seq: " << std::setw(6) << std::setfill('0') << metadata.sequence << " bytesused: "; + + unsigned int nplane = 0; + for (const FrameMetadata::Plane &plane : metadata.planes) + { + std::cout << plane.bytesused; + if (++nplane < metadata.planes.size()) std::cout << "/"; + } + + std::cout << std::endl; + +With the handling of this request complete, reuse the buffer by adding +it back to the request with its matching stream, and create a new +request using the ``createRequest`` function. + +.. code:: cpp + + request = camera->createRequest(); + if (!request) + { + std::cerr << "Can't create request" << std::endl; + return; + } + + for (auto it = buffers.begin(); it != buffers.end(); ++it) + { + Stream *stream = it->first; + FrameBuffer *buffer = it->second; + + request->addBuffer(stream, buffer); + } + + camera->queueRequest(request); + +Start the camera and event loop +------------------------------- + +If you build and run the application at this point, none of the code in +the slot method above runs. While most of the code to handle processing +camera data is in place, you need first to start the camera to begin +capturing frames and queuing requests to it. + +.. code:: cpp + + camera->start(); + for (Request *request : requests) + camera->queueRequest(request); + +To emit signals that slots can respond to, your application needs an +event loop. You can use the ``EventDispatcher`` class as an event loop +for your application to listen to signals from resources libcamera +handles. + +The libcamera library does this by creating instances of the +``EventNotifier`` class, which models a file descriptor event source an +application can monitor and registers them with the ``EventDispatcher``. +Whenever the ``EventDispatcher`` detects an event it is monitoring, and +it emits an ``EventNotifier::activated signal``. The ``Timer`` class to +control the length event loops run for that you can register with a +dispatcher with the ``registerTimer`` function. + +The code below creates a new instance of the ``EventDispatcher`` class, +adds it to the camera manager, creates a timer to run for 3 seconds, and +during the length of that timer, the ``EventDispatcher`` processes +events that occur, and calls the relevant signals. + +.. code:: cpp + + EventDispatcher *dispatcher = cm->eventDispatcher(); + Timer timer; + timer.start(3000); + while (timer.isRunning()) + dispatcher->processEvents(); + +Clean up and stop application +----------------------------- + +The application is now finished with the camera and the resources the +camera uses, so you need to do the following: + +- stop the camera +- free the stream from the ``FrameBufferAllocator`` +- delete the ``FrameBufferAllocator`` +- release the lock on the camera and reset the pointer to it +- stop the camera manager +- exit the application + +.. code:: cpp + + camera->stop(); + allocator->free(stream); + delete allocator; + camera->release(); + camera.reset(); + cm->stop(); + + return 0; + +Conclusion +----------