[libcamera-devel,v2,1/6] src: raspberrypi: Pass the drop frame count in start, not configure

10598 diff mbox series

The number of frames to drop (not display) is passed back now from the
start method, not configure. This means applications have a chance to
set fixed exposure/gain before starting the camera and this can affect
the frame drop count that is returned.

Note how we need to be able to tell the very first time we start the
camera from subsequent restarts, hence addition of the "firstStart_"
flag.

Both the IPA implementation file and the pipeline handler need
matching modifications.

Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
Reviewed-by: Naushir Patuck <naush@raspberrypi.com>
---
 src/ipa/raspberrypi/raspberrypi.cpp           | 45 ++++++++++---------
 .../pipeline/raspberrypi/raspberrypi.cpp      | 25 ++++++-----
 2 files changed, 38 insertions(+), 32 deletions(-)

Hi David,

Thank you for the patch.

On Mon, Dec 07, 2020 at 06:01:16PM +0000, David Plowman wrote:
> The number of frames to drop (not display) is passed back now from the
> start method, not configure. This means applications have a chance to
> set fixed exposure/gain before starting the camera and this can affect
> the frame drop count that is returned.
> 
> Note how we need to be able to tell the very first time we start the
> camera from subsequent restarts, hence addition of the "firstStart_"
> flag.

firstStart_ will stay set for the lifetime of the Camera object, as the
IPA interface lifetime matches it. That's probably too long, we should
at least reset if it the application closes the camera, but that's not
communicated to the IPA at the moment. Maybe an alternate mechanism,
such as a timeout, would be better. There's no need to address it now,
but I'm fairly sure the question will come back on the table at some
point :-)

> Both the IPA implementation file and the pipeline handler need
> matching modifications.
> 
> Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
> Reviewed-by: Naushir Patuck <naush@raspberrypi.com>
> ---
>  src/ipa/raspberrypi/raspberrypi.cpp           | 45 ++++++++++---------
>  .../pipeline/raspberrypi/raspberrypi.cpp      | 25 ++++++-----
>  2 files changed, 38 insertions(+), 32 deletions(-)
> 
> diff --git a/src/ipa/raspberrypi/raspberrypi.cpp b/src/ipa/raspberrypi/raspberrypi.cpp
> index 2027f1c0..0e89af00 100644
> --- a/src/ipa/raspberrypi/raspberrypi.cpp
> +++ b/src/ipa/raspberrypi/raspberrypi.cpp
> @@ -67,7 +67,7 @@ public:
>  	IPARPi()
>  		: lastMode_({}), controller_(), controllerInit_(false),
>  		  frameCount_(0), checkCount_(0), mistrustCount_(0),
> -		  lsTable_(nullptr)
> +		  lsTable_(nullptr), firstStart_(true)
>  	{
>  	}
>  
> @@ -145,6 +145,9 @@ private:
>  	/* LS table allocation passed in from the pipeline handler. */
>  	FileDescriptor lsTableHandle_;
>  	void *lsTable_;
> +
> +	/* Distinguish the first camera start from others. */
> +	bool firstStart_;
>  };
>  
>  int IPARPi::init(const IPASettings &settings)
> @@ -179,6 +182,27 @@ int IPARPi::start(const IPAOperationData &ipaConfig, IPAOperationData *result)
>  		result->operation |= RPi::IPA_CONFIG_SENSOR;
>  	}
>  
> +	/*
> +	 * Initialise frame counts, and decide how many frames must be hidden or
> +	 *"mistrusted", which depends on whether this is a startup from cold,

Missing space before "mistrusted".

Also not something to be addressed now: we'll likely move the
hide/mistrust count to a common helper in the future, to be shared by
IPA modules. As part of that effort, I think we'll try to formally
define what "hide" and "mistrust" means, and possibly rename those
terms. Do I understand correctly that "hide" refers to the number of
frames that are generated with incorrect parameters (mostly exposure
time I suppose), while "mistrust" is the number of frames generated with
incorrect metadata but without the frame itself being incorrect ?

I understand that the number of frames to hide is different as startup
and mode switch, as the IPA starts without context in the first case.
I'm however a bit puzzled as to why the mistrust count is different
between startup and mode switch. It seems to be

- 1 for startup and 0 for mode switch by default
- 2 in both cases for the OV5647
- 1 in both cases for the IMX219

The latter has a comment that the sensor sometimes returns incorrect
metadata on mode switch but not on startup, but the implementation for
IMX219 doesn't override MistrustFramesStartup(), and this uses a value
of 1. I wonder if this means the possible incorrect metadata at startup
has then just never been noticed, and if MistrustFramesStartup() and
MistrustFramesModeSwitch() should be merged into a single function.

These are not issues we need to address as part of this series, so

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> +	 * or merely a mode switch in a running system.
> +	 */
> +	frameCount_ = 0;
> +	checkCount_ = 0;
> +	unsigned int dropFrame = 0;
> +	if (firstStart_) {
> +		dropFrame = helper_->HideFramesStartup();
> +		mistrustCount_ = helper_->MistrustFramesStartup();
> +	} else {
> +		dropFrame = helper_->HideFramesModeSwitch();
> +		mistrustCount_ = helper_->MistrustFramesModeSwitch();
> +	}
> +
> +	result->data.push_back(dropFrame);
> +	result->operation |= RPi::IPA_CONFIG_DROP_FRAMES;
> +
> +	firstStart_ = false;
> +
>  	return 0;
>  }
>  
> @@ -293,25 +317,6 @@ void IPARPi::configure(const CameraSensorInfo &sensorInfo,
>  	/* Pass the camera mode to the CamHelper to setup algorithms. */
>  	helper_->SetCameraMode(mode_);
>  
> -	/*
> -	 * Initialise frame counts, and decide how many frames must be hidden or
> -	 *"mistrusted", which depends on whether this is a startup from cold,
> -	 * or merely a mode switch in a running system.
> -	 */
> -	frameCount_ = 0;
> -	checkCount_ = 0;
> -	unsigned int dropFrame = 0;
> -	if (controllerInit_) {
> -		dropFrame = helper_->HideFramesModeSwitch();
> -		mistrustCount_ = helper_->MistrustFramesModeSwitch();
> -	} else {
> -		dropFrame = helper_->HideFramesStartup();
> -		mistrustCount_ = helper_->MistrustFramesStartup();
> -	}
> -
> -	result->data.push_back(dropFrame);
> -	result->operation |= RPi::IPA_CONFIG_DROP_FRAMES;
> -
>  	if (!controllerInit_) {
>  		/* Load the tuning file for this sensor. */
>  		controller_.Read(tuningFile_.c_str());
> diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
> index 89a44763..5ae56628 100644
> --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
> +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
> @@ -745,13 +745,6 @@ int PipelineHandlerRPi::start(Camera *camera, [[maybe_unused]] ControlList *cont
>  		return ret;
>  	}
>  
> -	ret = queueAllBuffers(camera);
> -	if (ret) {
> -		LOG(RPI, Error) << "Failed to queue buffers";
> -		stop(camera);
> -		return ret;
> -	}
> -
>  	/* Check if a ScalerCrop control was specified. */
>  	if (controls)
>  		data->applyScalerCrop(*controls);
> @@ -778,6 +771,19 @@ int PipelineHandlerRPi::start(Camera *camera, [[maybe_unused]] ControlList *cont
>  			LOG(RPI, Error) << "V4L2 staggered set failed";
>  	}
>  
> +	if (result.operation & RPi::IPA_CONFIG_DROP_FRAMES) {
> +		/* Configure the number of dropped frames required on startup. */
> +		data->dropFrameCount_ = result.data[0];
> +	}
> +
> +	/* We need to set the dropFrameCount_ before queueing buffers. */
> +	ret = queueAllBuffers(camera);
> +	if (ret) {
> +		LOG(RPI, Error) << "Failed to queue buffers";
> +		stop(camera);
> +		return ret;
> +	}
> +
>  	/*
>  	 * IPA configure may have changed the sensor flips - hence the bayer
>  	 * order. Get the sensor format and set the ISP input now.
> @@ -1231,11 +1237,6 @@ int RPiCameraData::configureIPA(const CameraConfiguration *config)
>  			LOG(RPI, Error) << "V4L2 staggered set failed";
>  	}
>  
> -	if (result.operation & RPi::IPA_CONFIG_DROP_FRAMES) {
> -		/* Configure the number of dropped frames required on startup. */
> -		dropFrameCount_ = result.data[resultIdx++];
> -	}
> -
>  	/*
>  	 * Configure the H/V flip controls based on the combination of
>  	 * the sensor and user transform.