[libcamera-devel,v2,08/10] libcamera: pipeline: Add a simple pipeline handler

3125 diff mbox series

From: Martijn Braam <martijn@brixit.nl>

This new pipeline handler aims at supporting any simple device without
requiring any device-specific code. Simple devices are currently defined
as a graph made of one or multiple camera sensors and a single video
node, with each sensor connected to the video node through a linear
pipeline.

The simple pipeline handler will automatically parse the media graph,
enumerate sensors, build supported stream configurations, and configure
the pipeline, without any device-specific knowledge. It doesn't support
configuration of any processing in the pipeline at the moment, but may
be extended to support simple processing such as format conversion or
scaling in the future.

The only device-specific information in the pipeline handler is the list
of supported drivers, required for device matching. We may be able to
remove this in the future by matching with the simple pipeline handler
as a last resort option, after all other pipeline handlers have been
tried.

Signed-off-by: Martijn Braam <martijn@brixit.nl>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
---
Changes since v1:

- Rebase on top of buffer API rework
- Expose stream formats
- Rework camera data config
---
 src/libcamera/pipeline/meson.build        |   1 +
 src/libcamera/pipeline/simple/meson.build |   3 +
 src/libcamera/pipeline/simple/simple.cpp  | 699 ++++++++++++++++++++++
 3 files changed, 703 insertions(+)
 create mode 100644 src/libcamera/pipeline/simple/meson.build
 create mode 100644 src/libcamera/pipeline/simple/simple.cpp

Hi Laurent,

On Mon, Mar 16, 2020 at 11:43:08PM +0200, Laurent Pinchart wrote:
> From: Martijn Braam <martijn@brixit.nl>
>
> This new pipeline handler aims at supporting any simple device without
> requiring any device-specific code. Simple devices are currently defined
> as a graph made of one or multiple camera sensors and a single video
> node, with each sensor connected to the video node through a linear
> pipeline.
>
> The simple pipeline handler will automatically parse the media graph,
> enumerate sensors, build supported stream configurations, and configure
> the pipeline, without any device-specific knowledge. It doesn't support
> configuration of any processing in the pipeline at the moment, but may
> be extended to support simple processing such as format conversion or
> scaling in the future.
>
> The only device-specific information in the pipeline handler is the list
> of supported drivers, required for device matching. We may be able to
> remove this in the future by matching with the simple pipeline handler
> as a last resort option, after all other pipeline handlers have been
> tried.
>
> Signed-off-by: Martijn Braam <martijn@brixit.nl>
> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> ---
> Changes since v1:
>
> - Rebase on top of buffer API rework
> - Expose stream formats
> - Rework camera data config
> ---
>  src/libcamera/pipeline/meson.build        |   1 +
>  src/libcamera/pipeline/simple/meson.build |   3 +
>  src/libcamera/pipeline/simple/simple.cpp  | 699 ++++++++++++++++++++++
>  3 files changed, 703 insertions(+)
>  create mode 100644 src/libcamera/pipeline/simple/meson.build
>  create mode 100644 src/libcamera/pipeline/simple/simple.cpp
>
> diff --git a/src/libcamera/pipeline/meson.build b/src/libcamera/pipeline/meson.build
> index 0d466225a72e..606ba31a0319 100644
> --- a/src/libcamera/pipeline/meson.build
> +++ b/src/libcamera/pipeline/meson.build
> @@ -5,3 +5,4 @@ libcamera_sources += files([
>
>  subdir('ipu3')
>  subdir('rkisp1')
> +subdir('simple')
> diff --git a/src/libcamera/pipeline/simple/meson.build b/src/libcamera/pipeline/simple/meson.build
> new file mode 100644
> index 000000000000..4945a3e173cf
> --- /dev/null
> +++ b/src/libcamera/pipeline/simple/meson.build
> @@ -0,0 +1,3 @@
> +libcamera_sources += files([
> +    'simple.cpp',

Is this worth its own subdirectory or can it live at top level like
uvc and vimc ?

> +])
> diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp
> new file mode 100644
> index 000000000000..2126799c54eb
> --- /dev/null
> +++ b/src/libcamera/pipeline/simple/simple.cpp
> @@ -0,0 +1,699 @@
> +/* SPDX-License-Identifier: LGPL-2.1-or-later */
> +/*
> + * Copyright (C) 2020, Laurent Pinchart
> + * Copyright (C) 2019, Martijn Braam
> + *
> + * simple.cpp - Pipeline handler for simple pipelines
> + */
> +
> +#include <algorithm>
> +#include <iterator>
> +#include <list>
> +#include <map>
> +#include <memory>
> +#include <set>
> +#include <string>
> +#include <string.h>
> +#include <utility>
> +#include <vector>
> +
> +#include <linux/media-bus-format.h>
> +
> +#include <libcamera/camera.h>
> +#include <libcamera/request.h>
> +#include <libcamera/stream.h>
> +
> +#include "camera_sensor.h"
> +#include "device_enumerator.h"
> +#include "log.h"
> +#include "media_device.h"
> +#include "pipeline_handler.h"
> +#include "v4l2_subdevice.h"
> +#include "v4l2_videodevice.h"
> +
> +namespace libcamera {
> +
> +LOG_DEFINE_CATEGORY(SimplePipeline)
> +
> +class SimplePipelineHandler;
> +
> +class SimpleCameraData : public CameraData
> +{
> +public:
> +	SimpleCameraData(PipelineHandler *pipe, MediaEntity *sensor,
> +			 MediaEntity *video);
> +
> +	bool isValid() const { return sensor_ != nullptr; }
> +	std::set<Stream *> streams() { return { &stream_ }; }
> +
> +	int init();
> +	int setupLinks();
> +	int setupFormats(V4L2SubdeviceFormat *format,
> +			 V4L2Subdevice::Whence whence);
> +
> +	struct Entity {
> +		MediaEntity *entity;
> +		MediaLink *link;
> +	};
> +
> +	struct Configuration {
> +		uint32_t code;
> +		PixelFormat pixelFormat;
> +		Size size;
> +	};
> +
> +	Stream stream_;
> +	std::unique_ptr<CameraSensor> sensor_;
> +	std::list<Entity> entities_;
> +
> +	std::vector<Configuration> configs_;
> +	std::map<PixelFormat, Configuration> formats_;
> +};
> +
> +class SimpleCameraConfiguration : public CameraConfiguration
> +{
> +public:
> +	SimpleCameraConfiguration(Camera *camera, SimpleCameraData *data);
> +
> +	Status validate() override;
> +
> +	const V4L2SubdeviceFormat &sensorFormat() { return sensorFormat_; }
> +
> +private:
> +	/*
> +	 * The SimpleCameraData instance is guaranteed to be valid as long as
> +	 * the corresponding Camera instance is valid. In order to borrow a
> +	 * reference to the camera data, store a new reference to the camera.
> +	 */
> +	std::shared_ptr<Camera> camera_;
> +	const SimpleCameraData *data_;
> +
> +	V4L2SubdeviceFormat sensorFormat_;
> +};
> +
> +class SimplePipelineHandler : public PipelineHandler
> +{
> +public:
> +	SimplePipelineHandler(CameraManager *manager);
> +	~SimplePipelineHandler();
> +
> +	CameraConfiguration *generateConfiguration(Camera *camera,
> +						   const StreamRoles &roles) override;
> +	int configure(Camera *camera, CameraConfiguration *config) override;
> +
> +	int exportFrameBuffers(Camera *camera, Stream *stream,
> +			       std::vector<std::unique_ptr<FrameBuffer>> *buffers) override;
> +
> +	int start(Camera *camera) override;
> +	void stop(Camera *camera) override;
> +
> +	bool match(DeviceEnumerator *enumerator) override;
> +
> +	V4L2VideoDevice *video() { return video_; }
> +	V4L2Subdevice *subdev(const MediaEntity *entity);
> +
> +protected:
> +	int queueRequestDevice(Camera *camera, Request *request) override;
> +
> +private:
> +	SimpleCameraData *cameraData(const Camera *camera)
> +	{
> +		return static_cast<SimpleCameraData *>(
> +			PipelineHandler::cameraData(camera));
> +	}
> +
> +	int initLinks();
> +
> +	int createCamera(MediaEntity *sensor);
> +
> +	void bufferReady(FrameBuffer *buffer);
> +
> +	MediaDevice *media_;
> +	V4L2VideoDevice *video_;
> +	std::map<const MediaEntity *, V4L2Subdevice> subdevs_;
> +
> +	Camera *activeCamera_;
> +};
> +
> +/* -----------------------------------------------------------------------------
> + * Camera Data
> + */
> +SimpleCameraData::SimpleCameraData(PipelineHandler *pipe, MediaEntity *sensor,
> +				   MediaEntity *video)
> +	: CameraData(pipe)
> +{
> +	int ret;
> +
> +	/*
> +	 * Walk the pipeline towards the video node and store all entities
> +	 * along the way.
> +	 */
> +	MediaEntity *source = sensor;
> +
> +	while (source) {
> +		/* If we have reached the video node, we're done. */
> +		if (source == video)
> +			break;
> +
> +		/* Use the first output pad that has links. */
> +		MediaPad *sourcePad = nullptr;
> +		for (MediaPad *pad : source->pads()) {
> +			if ((pad->flags() & MEDIA_PAD_FL_SOURCE) &&
> +			    !pad->links().empty()) {
> +				sourcePad = pad;
> +				break;
> +			}
> +		}
> +
> +		if (!sourcePad)
> +			return;
> +
> +		/* Use the first link that isn't immutable and disabled. */
> +		MediaLink *sourceLink = nullptr;
> +		for (MediaLink *link : sourcePad->links()) {
> +			if ((link->flags() & MEDIA_LNK_FL_ENABLED) ||
> +			    !(link->flags() & MEDIA_LNK_FL_IMMUTABLE)) {
> +				sourceLink = link;
> +				break;
> +			}
> +		}
> +
> +		if (!sourceLink)
> +			return;
> +
> +		entities_.push_back({ source, sourceLink });
> +
> +		source = sourceLink->sink()->entity();
> +
> +		/* Avoid infinite loops. */

Is there a case for circular loops in media pipelines ?

> +		auto iter = std::find_if(entities_.begin(), entities_.end(),
> +					 [&](const Entity &entity) {
> +						 return entity.entity == source;
> +					 });
> +		if (iter != entities_.end()) {
> +			LOG(SimplePipeline, Info) << "Loop detected in pipeline";
> +			return;
> +		}
> +	}
> +
> +	/* We have a valid pipeline, create the camera sensor. */
> +	sensor_ = std::make_unique<CameraSensor>(sensor);
> +	ret = sensor_->init();
> +	if (ret) {
> +		sensor_.reset();
> +		return;
> +	}
> +}
> +
> +int SimpleCameraData::init()
> +{
> +	SimplePipelineHandler *pipe = static_cast<SimplePipelineHandler *>(pipe_);
> +	V4L2VideoDevice *video = pipe->video();
> +	int ret;
> +
> +	/*
> +	 * Enumerate the possible pipeline configurations. For each media bus
> +	 * format supported by the sensor, propagate the formats through the
> +	 * pipeline, and enumerate the corresponding possible V4L2 pixel
> +	 * formats on the video node.
> +	 */
> +	for (unsigned int code : sensor_->mbusCodes()) {
> +		V4L2SubdeviceFormat format{ code, sensor_->resolution() };

This (and in SimplePipeline::configure()) you seems to assume the max
sensor resolution is available for all mbus codes. I know the below
setupLinks() applies the format to the sensor, so size will be
adjusted, but I wonder if would not be worth associating in
Configuration the max sensor size associated with an mbus code to
avoid adjustments.

Also, speaking of it, isn't "Configuration" a too generic name ? I had
to look it up to see if it's a construct of this pipeline handler or a
general libcamera construct.

> +
> +		/*
> +		 * Setup links first as some subdev drivers take active links
> +		 * into account to propaget TRY formats. So is life :-(
> +		 */
> +		ret = setupLinks();
> +		if (ret < 0)
> +			return ret;
> +
> +		ret = setupFormats(&format, V4L2Subdevice::TryFormat);
> +		if (ret < 0)
> +			return ret;

SO for each supported mbus_code we go through a full pipeline
configuration. setupFormat() only fails when applying the format to
the sensor, so I wonder if a full pipeline config is required. Or
should you fail when applying formats in setupFormats to intermediate
entities as well ?

> +
> +		std::vector<unsigned int> formats =
> +			video->formats(format.mbus_code).formats();
> +
> +		LOG(SimplePipeline, Debug)
> +			<< "Adding configuration for " << format.size.toString()
> +			<< " in pixel formats [ "
> +			<< utils::join(formats, ", ",
> +				       [](unsigned int f) { return std::to_string(f); })
> +			<< " ]";
> +
> +		/*
> +		 * Store the configuration in the formats_ map, mapping
> +		 * PixelFormat to configuration. Any previously stored value is
> +		 * overwritten, as the pipeline handler currently doesn't care
> +		 * about how a particular PixelFormat is achieved.
> +		 */
> +		for (unsigned int v4l2Format : formats) {
> +			PixelFormat pixelFormat = video->toPixelFormat(v4l2Format);
> +			if (!pixelFormat)
> +				continue;
> +
> +			Configuration config;
> +			config.code = code;
> +			config.pixelFormat = pixelFormat;
> +			config.size = format.size;
> +
> +			formats_[pixelFormat] = config;
> +		}
> +	}
> +
> +	if (formats_.empty()) {
> +		LOG(SimplePipeline, Error) << "No valid configuration found";
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
> +int SimpleCameraData::setupLinks()
> +{
> +	int ret;
> +
> +	/*
> +	 * Configure all links along the pipeline. Some entities may not allow
> +	 * multiple sink links to be enabled together, even on different sink
> +	 * pads. We must thus start by disabling all sink links (but the one we
> +	 * want to enable) before enabling the pipeline link.
> +	 */
> +	for (SimpleCameraData::Entity &e : entities_) {
> +		for (MediaPad *pad : e.link->sink()->entity()->pads()) {
> +			for (MediaLink *link : pad->links()) {
> +				if (link == e.link)
> +					continue;
> +
> +				if ((link->flags() & MEDIA_LNK_FL_ENABLED) &&
> +				    !(link->flags() & MEDIA_LNK_FL_IMMUTABLE)) {
> +					ret = link->setEnabled(false);
> +					if (ret < 0)
> +						return ret;
> +				}
> +			}
> +		}
> +
> +		if (!(e.link->flags() & MEDIA_LNK_FL_ENABLED)) {
> +			ret = e.link->setEnabled(true);
> +			if (ret < 0)
> +				return ret;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +int SimpleCameraData::setupFormats(V4L2SubdeviceFormat *format,
> +				   V4L2Subdevice::Whence whence)

Not aligned to ( or is it my editor ?

> +{
> +	SimplePipelineHandler *pipe = static_cast<SimplePipelineHandler *>(pipe_);
> +	int ret;
> +
> +	/*
> +	 * Configure the format on the sensor output and propagate it through
> +	 * the pipeline.
nn> +	 */
> +	ret = sensor_->setFormat(format);
> +	if (ret < 0)
> +		return ret;
> +
> +	for (const Entity &e : entities_) {
> +		MediaLink *link = e.link;
> +		MediaPad *source = link->source();
> +		MediaPad *sink = link->sink();
> +
> +		if (source->entity() != sensor_->entity()) {
> +			V4L2Subdevice *subdev = pipe->subdev(source->entity());
> +			subdev->getFormat(source->index(), format, whence);
> +		}
> +
> +		if (sink->entity()->function() != MEDIA_ENT_F_IO_V4L) {
> +			V4L2Subdevice *subdev = pipe->subdev(sink->entity());
> +			subdev->setFormat(sink->index(), format, whence);

As reported above in SimpleCameraData::init(), should you check for return
value here ?

> +		}
> +

Brilliant, if not a bit terse to parse..
Maybe handling the sensor and the video node outside of the loop would
help making this a bit more clear, but it works and is compact, so
take this comment as I'm thinking out loud.

> +		LOG(SimplePipeline, Debug)
> +			<< "Link '" << source->entity()->name()
> +			<< "':" << source->index()
> +			<< " -> '" << sink->entity()->name()
> +			<< "':" << sink->index()
> +			<< " configured with format " << format->toString();
> +	}
> +
> +	return 0;
> +}
> +
> +/* -----------------------------------------------------------------------------
> + * Camera Configuration
> + */
> +
> +SimpleCameraConfiguration::SimpleCameraConfiguration(Camera *camera,
> +						     SimpleCameraData *data)
> +	: CameraConfiguration(), camera_(camera->shared_from_this()),
> +	  data_(data)
> +{
> +}
> +
> +CameraConfiguration::Status SimpleCameraConfiguration::validate()
> +{
> +	Status status = Valid;
> +
> +	if (config_.empty())
> +		return Invalid;
> +
> +	/* Cap the number of entries to the available streams. */
> +	if (config_.size() > 1) {
> +		config_.resize(1);
> +		status = Adjusted;
> +	}
> +
> +	StreamConfiguration &cfg = config_[0];
> +
> +	/* Adjust the pixel format. */
> +	auto it = data_->formats_.find(cfg.pixelFormat);
> +	if (it == data_->formats_.end())
> +		it = data_->formats_.begin();
> +
> +	PixelFormat pixelFormat = it->first;
> +	const SimpleCameraData::Configuration &pipeConfig = it->second;

nit: move this below

> +
> +	if (cfg.pixelFormat != pixelFormat) {
> +		LOG(SimplePipeline, Debug) << "Adjusting pixel format";
> +		cfg.pixelFormat = pixelFormat;
> +		status = Adjusted;
> +	}
> +
> +	if (cfg.size != pipeConfig.size) {
> +		LOG(SimplePipeline, Debug)
> +			<< "Adjusting size from " << cfg.size.toString()
> +			<< " to " << pipeConfig.size.toString();
> +		cfg.size = pipeConfig.size;
> +		status = Adjusted;
> +	}
> +
> +	cfg.bufferCount = 3;

Should't this come from the video device ?

> +
> +	return status;
> +}
> +
> +/* -----------------------------------------------------------------------------
> + * Pipeline Handler
> + */
> +
> +SimplePipelineHandler::SimplePipelineHandler(CameraManager *manager)
> +	: PipelineHandler(manager), video_(nullptr)
> +{
> +}
> +
> +SimplePipelineHandler::~SimplePipelineHandler()
> +{
> +	delete video_;
> +}
> +
> +CameraConfiguration *SimplePipelineHandler::generateConfiguration(Camera *camera,
> +								  const StreamRoles &roles)
> +{
> +	SimpleCameraData *data = cameraData(camera);
> +	CameraConfiguration *config =
> +		new SimpleCameraConfiguration(camera, data);
> +
> +	if (roles.empty())
> +		return config;

all roles are supported then ? There are not many ways around it if
not selecting the the stream with the max size for still capture and
defaulting to something smaller to viewfinder, but these are arbitrary
choices I guess

> +
> +	/* Create the formats map. */
> +	std::map<PixelFormat, std::vector<SizeRange>> formats;
> +	std::transform(data->formats_.begin(), data->formats_.end(),
> +		       std::inserter(formats, formats.end()),
> +		       [](const auto &format) -> decltype(formats)::value_type {
> +			       const PixelFormat &pixelFormat = format.first;
> +			       const Size &size = format.second.size;
> +			       return { pixelFormat, { size } };
> +		       });
> +
> +	/*
> +	 * Create the stream configuration. Take the first entry in the formats
> +	 * map as the default, for lack of a better option.
> +	 */

Which are not sorted, am I wrong ?

> +	StreamConfiguration cfg{ StreamFormats{ formats } };
> +	cfg.pixelFormat = formats.begin()->first;
> +	cfg.size = formats.begin()->second[0].max;
> +
> +	config->addConfiguration(cfg);
> +
> +	config->validate();
> +
> +	return config;
> +}
> +
> +int SimplePipelineHandler::configure(Camera *camera, CameraConfiguration *c)
> +{
> +	SimpleCameraConfiguration *config =
> +		static_cast<SimpleCameraConfiguration *>(c);
> +	SimpleCameraData *data = cameraData(camera);
> +	StreamConfiguration &cfg = config->at(0);
> +	int ret;
> +
> +	/*
> +	 * Configure links on the pipeline and propagate formats from the
> +	 * sensor to the video node.
> +	 */
> +	ret = data->setupLinks();
> +	if (ret < 0)
> +		return ret;
> +
> +	const SimpleCameraData::Configuration &pipeConfig =
> +		data->formats_[cfg.pixelFormat];
> +
> +	V4L2SubdeviceFormat format{ pipeConfig.code, data->sensor_->resolution() };
> +
> +	ret = data->setupFormats(&format, V4L2Subdevice::ActiveFormat);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* Configure the video node. */
> +	uint32_t fourcc = video_->toV4L2Fourcc(cfg.pixelFormat);
> +
> +	V4L2DeviceFormat outputFormat = {};
> +	outputFormat.fourcc = fourcc;
> +	outputFormat.size = cfg.size;
> +
> +	ret = video_->setFormat(&outputFormat);
> +	if (ret)
> +		return ret;
> +
> +	if (outputFormat.size != cfg.size || outputFormat.fourcc != fourcc) {
> +		LOG(SimplePipeline, Error)
> +			<< "Unable to configure capture in " << cfg.toString();
> +		return -EINVAL;
> +	}
> +
> +	cfg.setStream(&data->stream_);
> +
> +	return 0;
> +}
> +
> +int SimplePipelineHandler::exportFrameBuffers(Camera *camera, Stream *stream,
> +					      std::vector<std::unique_ptr<FrameBuffer>> *buffers)
> +{
> +	unsigned int count = stream->configuration().bufferCount;
> +
> +	return video_->exportBuffers(count, buffers);
> +}
> +
> +int SimplePipelineHandler::start(Camera *camera)
> +{
> +	SimpleCameraData *data = cameraData(camera);
> +	unsigned int count = data->stream_.configuration().bufferCount;
> +
> +	int ret = video_->importBuffers(count);
> +	if (ret < 0)
> +		return ret;
> +
> +	ret = video_->streamOn();
> +	if (ret < 0) {
> +		video_->releaseBuffers();
> +		return ret;
> +	}
> +
> +	activeCamera_ = camera;
> +
> +	return 0;
> +}
> +
> +void SimplePipelineHandler::stop(Camera *camera)
> +{
> +	video_->streamOff();
> +	video_->releaseBuffers();
> +	activeCamera_ = nullptr;
> +}
> +
> +int SimplePipelineHandler::queueRequestDevice(Camera *camera, Request *request)
> +{
> +	SimpleCameraData *data = cameraData(camera);
> +	Stream *stream = &data->stream_;

As in bufferReady we complete the request as soon as the buffer is
complete, should we make sure the request refers to a single stream ?

> +
> +	FrameBuffer *buffer = request->findBuffer(stream);
> +	if (!buffer) {
> +		LOG(SimplePipeline, Error)
> +			<< "Attempt to queue request with invalid stream";
> +		return -ENOENT;
> +	}
> +
> +	return video_->queueBuffer(buffer);
> +}
> +
> +/* -----------------------------------------------------------------------------
> + * Match and Setup
> + */
> +
> +bool SimplePipelineHandler::match(DeviceEnumerator *enumerator)
> +{
> +	static const char * const drivers[] = {
> +		"imx7-csi",
> +		"sun6i-csi",
> +	};
> +
> +	for (const char *driver : drivers) {
> +		DeviceMatch dm(driver);
> +		media_ = acquireMediaDevice(enumerator, dm);
> +		if (media_)
> +			break;
> +	}
> +
> +	if (!media_)
> +		return false;
> +
> +	/*
> +	 * Locate sensors and video nodes. We only support pipelines with at
> +	 * least one sensor and exactly one video captude node.
> +	 */
> +	std::vector<MediaEntity *> sensors;
> +	std::vector<MediaEntity *> videos;
> +
> +	for (MediaEntity *entity : media_->entities()) {
> +		switch (entity->function()) {
> +		case MEDIA_ENT_F_CAM_SENSOR:
> +			sensors.push_back(entity);
> +			break;
> +
> +		case MEDIA_ENT_F_IO_V4L:
> +			if (entity->pads().size() == 1 &&

Isn't this an arbitrary restinction ? Can't a video node a more than
one, maybe not connected, pad ?

> +			    (entity->pads()[0]->flags() & MEDIA_PAD_FL_SINK))
> +				videos.push_back(entity);
> +			break;
> +
> +		default:
> +			break;
> +		}
> +	}
> +
> +	if (sensors.empty()) {
> +		LOG(SimplePipeline, Info) << "No sensor found";
> +		return false;
> +	}
> +
> +	if (videos.size() != 1) {

Why a vector ? Cant you populate a single pointer and fail if it is
already != nullptr when walking device nodes ?

> +		LOG(SimplePipeline, Info)

s/Info/Error here and above

> +			<< "Pipeline with " << videos.size()
> +			<< " video capture nodes is not supported";
> +		return false;
> +	}
> +
> +	/* Locate and open the capture video node. */
> +	video_ = new V4L2VideoDevice(videos[0]);

If you remove the vector you can use video_ directly

> +	if (video_->open() < 0)
> +		return false;
> +
> +	if (video_->caps().isMultiplanar()) {

Why not mplane API support ?

> +		LOG(SimplePipeline, Info)

Error here as well ?

> +			<< "V4L2 multiplanar devices are not supported";
> +		return false;
> +	}
> +
> +	video_->bufferReady.connect(this, &SimplePipelineHandler::bufferReady);
> +
> +	/*
> +	 * Create one camera data instance for each sensor and gather all
> +	 * entities in all pipelines.
> +	 */

I might have missed the case for more sensor connected to the same
CSI-2 receiver..

> +	std::vector<std::unique_ptr<SimpleCameraData>> pipelines;
> +	std::set<MediaEntity *> entities;
> +
> +	pipelines.reserve(sensors.size());
> +
> +	for (MediaEntity *sensor : sensors) {
> +		std::unique_ptr<SimpleCameraData> data =
> +			std::make_unique<SimpleCameraData>(this, sensor,
> +							   videos[0]);
> +		if (!data->isValid()) {
> +			LOG(SimplePipeline, Info)

seems like using Info in place of Error is intentional

Quite some media entities, links, pads dancing in this pipeline :)
Good job!

Thanks
   j

> +				<< "No valid pipeline for sensor '"
> +				<< sensor->name() << "', skipping";
> +			continue;
> +		}
> +
> +		for (SimpleCameraData::Entity &entity : data->entities_)
> +			entities.insert(entity.entity);
> +
> +		pipelines.push_back(std::move(data));
> +	}
> +
> +	if (entities.empty())
> +		return false;
> +
> +	/* Create and open V4L2Subdev instances for all the entities. */
> +	for (MediaEntity *entity : entities) {
> +		auto elem = subdevs_.emplace(std::piecewise_construct,
> +					     std::forward_as_tuple(entity),
> +					     std::forward_as_tuple(entity));
> +		V4L2Subdevice *subdev = &elem.first->second;
> +		int ret = subdev->open();
> +		if (ret < 0) {
> +			LOG(SimplePipeline, Error)
> +				<< "Failed to open " << subdev->deviceNode()
> +				<< ": " << strerror(-ret);
> +			return false;
> +		}
> +	}
> +
> +	/* Initialize each pipeline and register a corresponding camera. */
> +	for (std::unique_ptr<SimpleCameraData> &data : pipelines) {
> +		int ret = data->init();
> +		if (ret < 0)
> +			continue;
> +
> +		std::shared_ptr<Camera> camera =
> +			Camera::create(this, data->sensor_->entity()->name(),
> +				       data->streams());
> +		registerCamera(std::move(camera), std::move(data));
> +	}
> +
> +	return true;
> +}
> +
> +V4L2Subdevice *SimplePipelineHandler::subdev(const MediaEntity *entity)
> +{
> +	auto iter = subdevs_.find(entity);
> +	if (iter == subdevs_.end())
> +		return nullptr;
> +
> +	return &iter->second;
> +}
> +
> +/* -----------------------------------------------------------------------------
> + * Buffer Handling
> + */
> +
> +void SimplePipelineHandler::bufferReady(FrameBuffer *buffer)
> +{
> +	ASSERT(activeCamera_);
> +	Request *request = buffer->request();
> +	completeBuffer(activeCamera_, request, buffer);
> +	completeRequest(activeCamera_, request);
> +}
> +
> +REGISTER_PIPELINE_HANDLER(SimplePipelineHandler);
> +
> +} /* namespace libcamera */
> --
> Regards,
>
> Laurent Pinchart
>
> _______________________________________________
> libcamera-devel mailing list
> libcamera-devel@lists.libcamera.org
> https://lists.libcamera.org/listinfo/libcamera-devel

Hi Jacopo,

On Tue, Mar 17, 2020 at 03:18:12PM +0100, Jacopo Mondi wrote:
> On Mon, Mar 16, 2020 at 11:43:08PM +0200, Laurent Pinchart wrote:
> > From: Martijn Braam <martijn@brixit.nl>
> >
> > This new pipeline handler aims at supporting any simple device without
> > requiring any device-specific code. Simple devices are currently defined
> > as a graph made of one or multiple camera sensors and a single video
> > node, with each sensor connected to the video node through a linear
> > pipeline.
> >
> > The simple pipeline handler will automatically parse the media graph,
> > enumerate sensors, build supported stream configurations, and configure
> > the pipeline, without any device-specific knowledge. It doesn't support
> > configuration of any processing in the pipeline at the moment, but may
> > be extended to support simple processing such as format conversion or
> > scaling in the future.
> >
> > The only device-specific information in the pipeline handler is the list
> > of supported drivers, required for device matching. We may be able to
> > remove this in the future by matching with the simple pipeline handler
> > as a last resort option, after all other pipeline handlers have been
> > tried.
> >
> > Signed-off-by: Martijn Braam <martijn@brixit.nl>
> > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > ---
> > Changes since v1:
> >
> > - Rebase on top of buffer API rework
> > - Expose stream formats
> > - Rework camera data config
> > ---
> >  src/libcamera/pipeline/meson.build        |   1 +
> >  src/libcamera/pipeline/simple/meson.build |   3 +
> >  src/libcamera/pipeline/simple/simple.cpp  | 699 ++++++++++++++++++++++
> >  3 files changed, 703 insertions(+)
> >  create mode 100644 src/libcamera/pipeline/simple/meson.build
> >  create mode 100644 src/libcamera/pipeline/simple/simple.cpp
> >
> > diff --git a/src/libcamera/pipeline/meson.build b/src/libcamera/pipeline/meson.build
> > index 0d466225a72e..606ba31a0319 100644
> > --- a/src/libcamera/pipeline/meson.build
> > +++ b/src/libcamera/pipeline/meson.build
> > @@ -5,3 +5,4 @@ libcamera_sources += files([
> >
> >  subdir('ipu3')
> >  subdir('rkisp1')
> > +subdir('simple')
> > diff --git a/src/libcamera/pipeline/simple/meson.build b/src/libcamera/pipeline/simple/meson.build
> > new file mode 100644
> > index 000000000000..4945a3e173cf
> > --- /dev/null
> > +++ b/src/libcamera/pipeline/simple/meson.build
> > @@ -0,0 +1,3 @@
> > +libcamera_sources += files([
> > +    'simple.cpp',
> 
> Is this worth its own subdirectory or can it live at top level like
> uvc and vimc ?

See patch 09/10 :-)

I think UVC and vimc should move to subdirectories anyway, one
subdirectory per pipeline handler is cleaner in my opinion. I'll send a
patch to propose that.

> > +])
> > diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp
> > new file mode 100644
> > index 000000000000..2126799c54eb
> > --- /dev/null
> > +++ b/src/libcamera/pipeline/simple/simple.cpp
> > @@ -0,0 +1,699 @@
> > +/* SPDX-License-Identifier: LGPL-2.1-or-later */
> > +/*
> > + * Copyright (C) 2020, Laurent Pinchart
> > + * Copyright (C) 2019, Martijn Braam
> > + *
> > + * simple.cpp - Pipeline handler for simple pipelines
> > + */
> > +
> > +#include <algorithm>
> > +#include <iterator>
> > +#include <list>
> > +#include <map>
> > +#include <memory>
> > +#include <set>
> > +#include <string>
> > +#include <string.h>
> > +#include <utility>
> > +#include <vector>
> > +
> > +#include <linux/media-bus-format.h>
> > +
> > +#include <libcamera/camera.h>
> > +#include <libcamera/request.h>
> > +#include <libcamera/stream.h>
> > +
> > +#include "camera_sensor.h"
> > +#include "device_enumerator.h"
> > +#include "log.h"
> > +#include "media_device.h"
> > +#include "pipeline_handler.h"
> > +#include "v4l2_subdevice.h"
> > +#include "v4l2_videodevice.h"
> > +
> > +namespace libcamera {
> > +
> > +LOG_DEFINE_CATEGORY(SimplePipeline)
> > +
> > +class SimplePipelineHandler;
> > +
> > +class SimpleCameraData : public CameraData
> > +{
> > +public:
> > +	SimpleCameraData(PipelineHandler *pipe, MediaEntity *sensor,
> > +			 MediaEntity *video);
> > +
> > +	bool isValid() const { return sensor_ != nullptr; }
> > +	std::set<Stream *> streams() { return { &stream_ }; }
> > +
> > +	int init();
> > +	int setupLinks();
> > +	int setupFormats(V4L2SubdeviceFormat *format,
> > +			 V4L2Subdevice::Whence whence);
> > +
> > +	struct Entity {
> > +		MediaEntity *entity;
> > +		MediaLink *link;
> > +	};
> > +
> > +	struct Configuration {
> > +		uint32_t code;
> > +		PixelFormat pixelFormat;
> > +		Size size;
> > +	};
> > +
> > +	Stream stream_;
> > +	std::unique_ptr<CameraSensor> sensor_;
> > +	std::list<Entity> entities_;
> > +
> > +	std::vector<Configuration> configs_;
> > +	std::map<PixelFormat, Configuration> formats_;
> > +};
> > +
> > +class SimpleCameraConfiguration : public CameraConfiguration
> > +{
> > +public:
> > +	SimpleCameraConfiguration(Camera *camera, SimpleCameraData *data);
> > +
> > +	Status validate() override;
> > +
> > +	const V4L2SubdeviceFormat &sensorFormat() { return sensorFormat_; }
> > +
> > +private:
> > +	/*
> > +	 * The SimpleCameraData instance is guaranteed to be valid as long as
> > +	 * the corresponding Camera instance is valid. In order to borrow a
> > +	 * reference to the camera data, store a new reference to the camera.
> > +	 */
> > +	std::shared_ptr<Camera> camera_;
> > +	const SimpleCameraData *data_;
> > +
> > +	V4L2SubdeviceFormat sensorFormat_;
> > +};
> > +
> > +class SimplePipelineHandler : public PipelineHandler
> > +{
> > +public:
> > +	SimplePipelineHandler(CameraManager *manager);
> > +	~SimplePipelineHandler();
> > +
> > +	CameraConfiguration *generateConfiguration(Camera *camera,
> > +						   const StreamRoles &roles) override;
> > +	int configure(Camera *camera, CameraConfiguration *config) override;
> > +
> > +	int exportFrameBuffers(Camera *camera, Stream *stream,
> > +			       std::vector<std::unique_ptr<FrameBuffer>> *buffers) override;
> > +
> > +	int start(Camera *camera) override;
> > +	void stop(Camera *camera) override;
> > +
> > +	bool match(DeviceEnumerator *enumerator) override;
> > +
> > +	V4L2VideoDevice *video() { return video_; }
> > +	V4L2Subdevice *subdev(const MediaEntity *entity);
> > +
> > +protected:
> > +	int queueRequestDevice(Camera *camera, Request *request) override;
> > +
> > +private:
> > +	SimpleCameraData *cameraData(const Camera *camera)
> > +	{
> > +		return static_cast<SimpleCameraData *>(
> > +			PipelineHandler::cameraData(camera));
> > +	}
> > +
> > +	int initLinks();
> > +
> > +	int createCamera(MediaEntity *sensor);
> > +
> > +	void bufferReady(FrameBuffer *buffer);
> > +
> > +	MediaDevice *media_;
> > +	V4L2VideoDevice *video_;
> > +	std::map<const MediaEntity *, V4L2Subdevice> subdevs_;
> > +
> > +	Camera *activeCamera_;
> > +};
> > +
> > +/* -----------------------------------------------------------------------------
> > + * Camera Data
> > + */
> > +SimpleCameraData::SimpleCameraData(PipelineHandler *pipe, MediaEntity *sensor,
> > +				   MediaEntity *video)
> > +	: CameraData(pipe)
> > +{
> > +	int ret;
> > +
> > +	/*
> > +	 * Walk the pipeline towards the video node and store all entities
> > +	 * along the way.
> > +	 */
> > +	MediaEntity *source = sensor;
> > +
> > +	while (source) {
> > +		/* If we have reached the video node, we're done. */
> > +		if (source == video)
> > +			break;
> > +
> > +		/* Use the first output pad that has links. */
> > +		MediaPad *sourcePad = nullptr;
> > +		for (MediaPad *pad : source->pads()) {
> > +			if ((pad->flags() & MEDIA_PAD_FL_SOURCE) &&
> > +			    !pad->links().empty()) {
> > +				sourcePad = pad;
> > +				break;
> > +			}
> > +		}
> > +
> > +		if (!sourcePad)
> > +			return;
> > +
> > +		/* Use the first link that isn't immutable and disabled. */
> > +		MediaLink *sourceLink = nullptr;
> > +		for (MediaLink *link : sourcePad->links()) {
> > +			if ((link->flags() & MEDIA_LNK_FL_ENABLED) ||
> > +			    !(link->flags() & MEDIA_LNK_FL_IMMUTABLE)) {
> > +				sourceLink = link;
> > +				break;
> > +			}
> > +		}
> > +
> > +		if (!sourceLink)
> > +			return;
> > +
> > +		entities_.push_back({ source, sourceLink });
> > +
> > +		source = sourceLink->sink()->entity();
> > +
> > +		/* Avoid infinite loops. */
> 
> Is there a case for circular loops in media pipelines ?

Not that I know of, but as we don't know what driver we'll deal with, I
thought it would be better to print an error message than looping until
the stack overflows. Do you think that's overkill ?

> > +		auto iter = std::find_if(entities_.begin(), entities_.end(),
> > +					 [&](const Entity &entity) {
> > +						 return entity.entity == source;
> > +					 });
> > +		if (iter != entities_.end()) {
> > +			LOG(SimplePipeline, Info) << "Loop detected in pipeline";
> > +			return;
> > +		}
> > +	}
> > +
> > +	/* We have a valid pipeline, create the camera sensor. */
> > +	sensor_ = std::make_unique<CameraSensor>(sensor);
> > +	ret = sensor_->init();
> > +	if (ret) {
> > +		sensor_.reset();
> > +		return;
> > +	}
> > +}
> > +
> > +int SimpleCameraData::init()
> > +{
> > +	SimplePipelineHandler *pipe = static_cast<SimplePipelineHandler *>(pipe_);
> > +	V4L2VideoDevice *video = pipe->video();
> > +	int ret;
> > +
> > +	/*
> > +	 * Enumerate the possible pipeline configurations. For each media bus
> > +	 * format supported by the sensor, propagate the formats through the
> > +	 * pipeline, and enumerate the corresponding possible V4L2 pixel
> > +	 * formats on the video node.
> > +	 */
> > +	for (unsigned int code : sensor_->mbusCodes()) {
> > +		V4L2SubdeviceFormat format{ code, sensor_->resolution() };
> 
> This (and in SimplePipeline::configure()) you seems to assume the max
> sensor resolution is available for all mbus codes. I know the below
> setupLinks() applies the format to the sensor, so size will be
> adjusted, but I wonder if would not be worth associating in
> Configuration the max sensor size associated with an mbus code to
> avoid adjustments.

It's the usual case. I know some sensors could implement this
differently, but I think that's room for future improvements. The simple
pipeline handler is nowhere close to be 100% complete.

> Also, speaking of it, isn't "Configuration" a too generic name ? I had
> to look it up to see if it's a construct of this pipeline handler or a
> general libcamera construct.

I'm not too fond of it either, so if you have a better alternative, I
can change it. I didn't want to go for SimpleConfiguration though, as
the name is too long, and sounds awkward as the Configuration struct is
already a member of SimpleCameraData.

> > +
> > +		/*
> > +		 * Setup links first as some subdev drivers take active links
> > +		 * into account to propaget TRY formats. So is life :-(
> > +		 */
> > +		ret = setupLinks();
> > +		if (ret < 0)
> > +			return ret;
> > +
> > +		ret = setupFormats(&format, V4L2Subdevice::TryFormat);
> > +		if (ret < 0)
> > +			return ret;
> 
> SO for each supported mbus_code we go through a full pipeline
> configuration. setupFormat() only fails when applying the format to
> the sensor, so I wonder if a full pipeline config is required. Or
> should you fail when applying formats in setupFormats to intermediate
> entities as well ?

It's not about failing, it's about propagating the format through the
pipeline. I'll fix setupFormats() to propagate errors though.

> > +
> > +		std::vector<unsigned int> formats =
> > +			video->formats(format.mbus_code).formats();
> > +
> > +		LOG(SimplePipeline, Debug)
> > +			<< "Adding configuration for " << format.size.toString()
> > +			<< " in pixel formats [ "
> > +			<< utils::join(formats, ", ",
> > +				       [](unsigned int f) { return std::to_string(f); })
> > +			<< " ]";
> > +
> > +		/*
> > +		 * Store the configuration in the formats_ map, mapping
> > +		 * PixelFormat to configuration. Any previously stored value is
> > +		 * overwritten, as the pipeline handler currently doesn't care
> > +		 * about how a particular PixelFormat is achieved.
> > +		 */
> > +		for (unsigned int v4l2Format : formats) {
> > +			PixelFormat pixelFormat = video->toPixelFormat(v4l2Format);
> > +			if (!pixelFormat)
> > +				continue;
> > +
> > +			Configuration config;
> > +			config.code = code;
> > +			config.pixelFormat = pixelFormat;
> > +			config.size = format.size;
> > +
> > +			formats_[pixelFormat] = config;
> > +		}
> > +	}
> > +
> > +	if (formats_.empty()) {
> > +		LOG(SimplePipeline, Error) << "No valid configuration found";
> > +		return -EINVAL;
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +int SimpleCameraData::setupLinks()
> > +{
> > +	int ret;
> > +
> > +	/*
> > +	 * Configure all links along the pipeline. Some entities may not allow
> > +	 * multiple sink links to be enabled together, even on different sink
> > +	 * pads. We must thus start by disabling all sink links (but the one we
> > +	 * want to enable) before enabling the pipeline link.
> > +	 */
> > +	for (SimpleCameraData::Entity &e : entities_) {
> > +		for (MediaPad *pad : e.link->sink()->entity()->pads()) {
> > +			for (MediaLink *link : pad->links()) {
> > +				if (link == e.link)
> > +					continue;
> > +
> > +				if ((link->flags() & MEDIA_LNK_FL_ENABLED) &&
> > +				    !(link->flags() & MEDIA_LNK_FL_IMMUTABLE)) {
> > +					ret = link->setEnabled(false);
> > +					if (ret < 0)
> > +						return ret;
> > +				}
> > +			}
> > +		}
> > +
> > +		if (!(e.link->flags() & MEDIA_LNK_FL_ENABLED)) {
> > +			ret = e.link->setEnabled(true);
> > +			if (ret < 0)
> > +				return ret;
> > +		}
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +int SimpleCameraData::setupFormats(V4L2SubdeviceFormat *format,
> > +				   V4L2Subdevice::Whence whence)
> 
> Not aligned to ( or is it my editor ?

It seems to be your editor.

> > +{
> > +	SimplePipelineHandler *pipe = static_cast<SimplePipelineHandler *>(pipe_);
> > +	int ret;
> > +
> > +	/*
> > +	 * Configure the format on the sensor output and propagate it through
> > +	 * the pipeline.
> > +	 */
> > +	ret = sensor_->setFormat(format);
> > +	if (ret < 0)
> > +		return ret;
> > +
> > +	for (const Entity &e : entities_) {
> > +		MediaLink *link = e.link;
> > +		MediaPad *source = link->source();
> > +		MediaPad *sink = link->sink();
> > +
> > +		if (source->entity() != sensor_->entity()) {
> > +			V4L2Subdevice *subdev = pipe->subdev(source->entity());
> > +			subdev->getFormat(source->index(), format, whence);
> > +		}
> > +
> > +		if (sink->entity()->function() != MEDIA_ENT_F_IO_V4L) {
> > +			V4L2Subdevice *subdev = pipe->subdev(sink->entity());
> > +			subdev->setFormat(sink->index(), format, whence);
> 
> As reported above in SimpleCameraData::init(), should you check for return
> value here ?

Yes, I'll change that.

> > +		}
> > +
> 
> Brilliant, if not a bit terse to parse..
> Maybe handling the sensor and the video node outside of the loop would
> help making this a bit more clear, but it works and is compact, so
> take this comment as I'm thinking out loud.

But they are outside of the loop already :-) The checks are meant to
ignore them in the loop. Each iteration needs to get the format on the
source and set it on the sink, for each link, with two exceptions:

- For the first link (where the source is the sensor), we have already
  just called setFormat() on the sensor, so there's no need to call
  getFormat(), that would be a waste of CPU time.

- For the last link, we don't want to set the format on the sink, as
  video nodes are not subdevs, they don't have pad ops.

> > +		LOG(SimplePipeline, Debug)
> > +			<< "Link '" << source->entity()->name()
> > +			<< "':" << source->index()
> > +			<< " -> '" << sink->entity()->name()
> > +			<< "':" << sink->index()
> > +			<< " configured with format " << format->toString();
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +/* -----------------------------------------------------------------------------
> > + * Camera Configuration
> > + */
> > +
> > +SimpleCameraConfiguration::SimpleCameraConfiguration(Camera *camera,
> > +						     SimpleCameraData *data)
> > +	: CameraConfiguration(), camera_(camera->shared_from_this()),
> > +	  data_(data)
> > +{
> > +}
> > +
> > +CameraConfiguration::Status SimpleCameraConfiguration::validate()
> > +{
> > +	Status status = Valid;
> > +
> > +	if (config_.empty())
> > +		return Invalid;
> > +
> > +	/* Cap the number of entries to the available streams. */
> > +	if (config_.size() > 1) {
> > +		config_.resize(1);
> > +		status = Adjusted;
> > +	}
> > +
> > +	StreamConfiguration &cfg = config_[0];
> > +
> > +	/* Adjust the pixel format. */
> > +	auto it = data_->formats_.find(cfg.pixelFormat);
> > +	if (it == data_->formats_.end())
> > +		it = data_->formats_.begin();
> > +
> > +	PixelFormat pixelFormat = it->first;
> > +	const SimpleCameraData::Configuration &pipeConfig = it->second;
> 
> nit: move this below

Done.

> > +
> > +	if (cfg.pixelFormat != pixelFormat) {
> > +		LOG(SimplePipeline, Debug) << "Adjusting pixel format";
> > +		cfg.pixelFormat = pixelFormat;
> > +		status = Adjusted;
> > +	}
> > +
> > +	if (cfg.size != pipeConfig.size) {
> > +		LOG(SimplePipeline, Debug)
> > +			<< "Adjusting size from " << cfg.size.toString()
> > +			<< " to " << pipeConfig.size.toString();
> > +		cfg.size = pipeConfig.size;
> > +		status = Adjusted;
> > +	}
> > +
> > +	cfg.bufferCount = 3;
> 
> Should't this come from the video device ?

Once V4L2 will give us the information, sure :-)

> > +
> > +	return status;
> > +}
> > +
> > +/* -----------------------------------------------------------------------------
> > + * Pipeline Handler
> > + */
> > +
> > +SimplePipelineHandler::SimplePipelineHandler(CameraManager *manager)
> > +	: PipelineHandler(manager), video_(nullptr)
> > +{
> > +}
> > +
> > +SimplePipelineHandler::~SimplePipelineHandler()
> > +{
> > +	delete video_;
> > +}
> > +
> > +CameraConfiguration *SimplePipelineHandler::generateConfiguration(Camera *camera,
> > +								  const StreamRoles &roles)
> > +{
> > +	SimpleCameraData *data = cameraData(camera);
> > +	CameraConfiguration *config =
> > +		new SimpleCameraConfiguration(camera, data);
> > +
> > +	if (roles.empty())
> > +		return config;
> 
> all roles are supported then ? There are not many ways around it if
> not selecting the the stream with the max size for still capture and
> defaulting to something smaller to viewfinder, but these are arbitrary
> choices I guess

There's a single stream, without scaling, so the role is a bit
irrelevant I think. Maybe we'll implement something smarter in the
future, maybe not.

> > +
> > +	/* Create the formats map. */
> > +	std::map<PixelFormat, std::vector<SizeRange>> formats;
> > +	std::transform(data->formats_.begin(), data->formats_.end(),
> > +		       std::inserter(formats, formats.end()),
> > +		       [](const auto &format) -> decltype(formats)::value_type {
> > +			       const PixelFormat &pixelFormat = format.first;
> > +			       const Size &size = format.second.size;
> > +			       return { pixelFormat, { size } };
> > +		       });
> > +
> > +	/*
> > +	 * Create the stream configuration. Take the first entry in the formats
> > +	 * map as the default, for lack of a better option.
> > +	 */
> 
> Which are not sorted, am I wrong ?

They're sorted by PixelFormat (std::map is ordered). This is purely
arbitrary, and I think we should improve that in the future. I'll add a
\todo.

> > +	StreamConfiguration cfg{ StreamFormats{ formats } };
> > +	cfg.pixelFormat = formats.begin()->first;
> > +	cfg.size = formats.begin()->second[0].max;
> > +
> > +	config->addConfiguration(cfg);
> > +
> > +	config->validate();
> > +
> > +	return config;
> > +}
> > +
> > +int SimplePipelineHandler::configure(Camera *camera, CameraConfiguration *c)
> > +{
> > +	SimpleCameraConfiguration *config =
> > +		static_cast<SimpleCameraConfiguration *>(c);
> > +	SimpleCameraData *data = cameraData(camera);
> > +	StreamConfiguration &cfg = config->at(0);
> > +	int ret;
> > +
> > +	/*
> > +	 * Configure links on the pipeline and propagate formats from the
> > +	 * sensor to the video node.
> > +	 */
> > +	ret = data->setupLinks();
> > +	if (ret < 0)
> > +		return ret;
> > +
> > +	const SimpleCameraData::Configuration &pipeConfig =
> > +		data->formats_[cfg.pixelFormat];
> > +
> > +	V4L2SubdeviceFormat format{ pipeConfig.code, data->sensor_->resolution() };
> > +
> > +	ret = data->setupFormats(&format, V4L2Subdevice::ActiveFormat);
> > +	if (ret < 0)
> > +		return ret;
> > +
> > +	/* Configure the video node. */
> > +	uint32_t fourcc = video_->toV4L2Fourcc(cfg.pixelFormat);
> > +
> > +	V4L2DeviceFormat outputFormat = {};
> > +	outputFormat.fourcc = fourcc;
> > +	outputFormat.size = cfg.size;
> > +
> > +	ret = video_->setFormat(&outputFormat);
> > +	if (ret)
> > +		return ret;
> > +
> > +	if (outputFormat.size != cfg.size || outputFormat.fourcc != fourcc) {
> > +		LOG(SimplePipeline, Error)
> > +			<< "Unable to configure capture in " << cfg.toString();
> > +		return -EINVAL;
> > +	}
> > +
> > +	cfg.setStream(&data->stream_);
> > +
> > +	return 0;
> > +}
> > +
> > +int SimplePipelineHandler::exportFrameBuffers(Camera *camera, Stream *stream,
> > +					      std::vector<std::unique_ptr<FrameBuffer>> *buffers)
> > +{
> > +	unsigned int count = stream->configuration().bufferCount;
> > +
> > +	return video_->exportBuffers(count, buffers);
> > +}
> > +
> > +int SimplePipelineHandler::start(Camera *camera)
> > +{
> > +	SimpleCameraData *data = cameraData(camera);
> > +	unsigned int count = data->stream_.configuration().bufferCount;
> > +
> > +	int ret = video_->importBuffers(count);
> > +	if (ret < 0)
> > +		return ret;
> > +
> > +	ret = video_->streamOn();
> > +	if (ret < 0) {
> > +		video_->releaseBuffers();
> > +		return ret;
> > +	}
> > +
> > +	activeCamera_ = camera;
> > +
> > +	return 0;
> > +}
> > +
> > +void SimplePipelineHandler::stop(Camera *camera)
> > +{
> > +	video_->streamOff();
> > +	video_->releaseBuffers();
> > +	activeCamera_ = nullptr;
> > +}
> > +
> > +int SimplePipelineHandler::queueRequestDevice(Camera *camera, Request *request)
> > +{
> > +	SimpleCameraData *data = cameraData(camera);
> > +	Stream *stream = &data->stream_;
> 
> As in bufferReady we complete the request as soon as the buffer is
> complete, should we make sure the request refers to a single stream ?

As we create a single stream, that's enforced by the Camera class and
the Request class. Camera::queueRequest() verifies that all buffers in
the request correspond to active streams for the camera, and the request
class stores buffers in a map indexed by Stream pointer, so we can't
have two buffers for the same stream.

> > +
> > +	FrameBuffer *buffer = request->findBuffer(stream);
> > +	if (!buffer) {
> > +		LOG(SimplePipeline, Error)
> > +			<< "Attempt to queue request with invalid stream";
> > +		return -ENOENT;
> > +	}
> > +
> > +	return video_->queueBuffer(buffer);
> > +}
> > +
> > +/* -----------------------------------------------------------------------------
> > + * Match and Setup
> > + */
> > +
> > +bool SimplePipelineHandler::match(DeviceEnumerator *enumerator)
> > +{
> > +	static const char * const drivers[] = {
> > +		"imx7-csi",
> > +		"sun6i-csi",
> > +	};
> > +
> > +	for (const char *driver : drivers) {
> > +		DeviceMatch dm(driver);
> > +		media_ = acquireMediaDevice(enumerator, dm);
> > +		if (media_)
> > +			break;
> > +	}
> > +
> > +	if (!media_)
> > +		return false;
> > +
> > +	/*
> > +	 * Locate sensors and video nodes. We only support pipelines with at
> > +	 * least one sensor and exactly one video captude node.
> > +	 */
> > +	std::vector<MediaEntity *> sensors;
> > +	std::vector<MediaEntity *> videos;
> > +
> > +	for (MediaEntity *entity : media_->entities()) {
> > +		switch (entity->function()) {
> > +		case MEDIA_ENT_F_CAM_SENSOR:
> > +			sensors.push_back(entity);
> > +			break;
> > +
> > +		case MEDIA_ENT_F_IO_V4L:
> > +			if (entity->pads().size() == 1 &&
> 
> Isn't this an arbitrary restinction ? Can't a video node a more than
> one, maybe not connected, pad ?

Not currently, no. And I want the rest of this code to be able to assume
that the video node has a single pad, hence the check here. We will lift
that restriction later if needed.

> > +			    (entity->pads()[0]->flags() & MEDIA_PAD_FL_SINK))
> > +				videos.push_back(entity);
> > +			break;
> > +
> > +		default:
> > +			break;
> > +		}
> > +	}
> > +
> > +	if (sensors.empty()) {
> > +		LOG(SimplePipeline, Info) << "No sensor found";
> > +		return false;
> > +	}
> > +
> > +	if (videos.size() != 1) {
> 
> Why a vector ? Cant you populate a single pointer and fail if it is
> already != nullptr when walking device nodes ?

I could, but I found it clearer this way, the check really says "is
there more than one video node ?", and goes at the end. Efficiency isn't
an issue, for all valid pipelines there will be a single video node, and
this isn't a hot path. Note that I will need to add at least a null
check after the loop to catch the size() == 0 case, even if I move the
size() > 1 case within the loop. I can change it if you think checking
in the loop is better.

> > +		LOG(SimplePipeline, Info)
> 
> s/Info/Error here and above

I used Info here as it may not be an error, it could just be that the
hardware isn't compatible with the simple pipeline handler. But maybe
the driver name shouldn't be listed in that case. I'll change to Error.

> > +			<< "Pipeline with " << videos.size()
> > +			<< " video capture nodes is not supported";
> > +		return false;
> > +	}
> > +
> > +	/* Locate and open the capture video node. */
> > +	video_ = new V4L2VideoDevice(videos[0]);
> 
> If you remove the vector you can use video_ directly
> 
> > +	if (video_->open() < 0)
> > +		return false;
> > +
> > +	if (video_->caps().isMultiplanar()) {
> 
> Why not mplane API support ?

It's just not supported yet. I'm sure there are lots of features that
will be added to this pipeline handler over time, this being one of
them. I wonder if it will become our most complex pipeline handler,
despite its name :-)

> > +		LOG(SimplePipeline, Info)
> 
> Error here as well ?
> 
> > +			<< "V4L2 multiplanar devices are not supported";
> > +		return false;
> > +	}
> > +
> > +	video_->bufferReady.connect(this, &SimplePipelineHandler::bufferReady);
> > +
> > +	/*
> > +	 * Create one camera data instance for each sensor and gather all
> > +	 * entities in all pipelines.
> > +	 */
> 
> I might have missed the case for more sensor connected to the same
> CSI-2 receiver..

Yes, there are platform with two sensors connected to the same CSI-2
receiver (or sometimes to two CSI-2 receivers that are then connected to
a video mux).

> > +	std::vector<std::unique_ptr<SimpleCameraData>> pipelines;
> > +	std::set<MediaEntity *> entities;
> > +
> > +	pipelines.reserve(sensors.size());
> > +
> > +	for (MediaEntity *sensor : sensors) {
> > +		std::unique_ptr<SimpleCameraData> data =
> > +			std::make_unique<SimpleCameraData>(this, sensor,
> > +							   videos[0]);
> > +		if (!data->isValid()) {
> > +			LOG(SimplePipeline, Info)
> 
> seems like using Info in place of Error is intentional
>
> Quite some media entities, links, pads dancing in this pipeline :)
> Good job!

Thanks :-) And thank you for the detailed review !

> > +				<< "No valid pipeline for sensor '"
> > +				<< sensor->name() << "', skipping";
> > +			continue;
> > +		}
> > +
> > +		for (SimpleCameraData::Entity &entity : data->entities_)
> > +			entities.insert(entity.entity);
> > +
> > +		pipelines.push_back(std::move(data));
> > +	}
> > +
> > +	if (entities.empty())
> > +		return false;
> > +
> > +	/* Create and open V4L2Subdev instances for all the entities. */
> > +	for (MediaEntity *entity : entities) {
> > +		auto elem = subdevs_.emplace(std::piecewise_construct,
> > +					     std::forward_as_tuple(entity),
> > +					     std::forward_as_tuple(entity));
> > +		V4L2Subdevice *subdev = &elem.first->second;
> > +		int ret = subdev->open();
> > +		if (ret < 0) {
> > +			LOG(SimplePipeline, Error)
> > +				<< "Failed to open " << subdev->deviceNode()
> > +				<< ": " << strerror(-ret);
> > +			return false;
> > +		}
> > +	}
> > +
> > +	/* Initialize each pipeline and register a corresponding camera. */
> > +	for (std::unique_ptr<SimpleCameraData> &data : pipelines) {
> > +		int ret = data->init();
> > +		if (ret < 0)
> > +			continue;
> > +
> > +		std::shared_ptr<Camera> camera =
> > +			Camera::create(this, data->sensor_->entity()->name(),
> > +				       data->streams());
> > +		registerCamera(std::move(camera), std::move(data));
> > +	}
> > +
> > +	return true;
> > +}
> > +
> > +V4L2Subdevice *SimplePipelineHandler::subdev(const MediaEntity *entity)
> > +{
> > +	auto iter = subdevs_.find(entity);
> > +	if (iter == subdevs_.end())
> > +		return nullptr;
> > +
> > +	return &iter->second;
> > +}
> > +
> > +/* -----------------------------------------------------------------------------
> > + * Buffer Handling
> > + */
> > +
> > +void SimplePipelineHandler::bufferReady(FrameBuffer *buffer)
> > +{
> > +	ASSERT(activeCamera_);
> > +	Request *request = buffer->request();
> > +	completeBuffer(activeCamera_, request, buffer);
> > +	completeRequest(activeCamera_, request);
> > +}
> > +
> > +REGISTER_PIPELINE_HANDLER(SimplePipelineHandler);
> > +
> > +} /* namespace libcamera */