[libcamera-devel,v5,2/6] pipeline: raspberrypi: Set the ISP Output1 to 1/4 resolution if unused
diff mbox series

Message ID 20210129151154.1051163-3-naush@raspberrypi.com
State Changes Requested
Headers show
Series
  • Raspberry Pi: Colour denoise
Related show

Commit Message

Naushir Patuck Jan. 29, 2021, 3:11 p.m. UTC 
In preparation for fast colour denoise, set the low resolution ISP
output stream (Output1) to a 1/4 resolution of the application requested
stream (Output0). This only happens if the application has not requested
an additional YUV or RGB stream.

We also constrain this 1/4 resolution to at most 1200 pixels in the
largest dimension to avoid being too taxing on memory usage and system
bandwidth.

Also switch the default StreamRole::VideoRecording to YUV420 to allow
fast colour denoise to run.

Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
---
 .../pipeline/raspberrypi/raspberrypi.cpp      | 35 ++++++++++++++++++-
 1 file changed, 34 insertions(+), 1 deletion(-)

Comments

Laurent Pinchart Feb. 4, 2021, 9:31 p.m. UTC | #1 
Hi Naush,

Thank you for the patch.

On Fri, Jan 29, 2021 at 03:11:50PM +0000, Naushir Patuck wrote:
> In preparation for fast colour denoise, set the low resolution ISP
> output stream (Output1) to a 1/4 resolution of the application requested
> stream (Output0). This only happens if the application has not requested
> an additional YUV or RGB stream.
> 
> We also constrain this 1/4 resolution to at most 1200 pixels in the
> largest dimension to avoid being too taxing on memory usage and system
> bandwidth.
> 
> Also switch the default StreamRole::VideoRecording to YUV420 to allow
> fast colour denoise to run.
> 
> Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
> Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> ---
>  .../pipeline/raspberrypi/raspberrypi.cpp      | 35 ++++++++++++++++++-
>  1 file changed, 34 insertions(+), 1 deletion(-)
> 
> diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
> index d9895c779725..fe4c75f09925 100644
> --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
> +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
> @@ -496,7 +496,7 @@ CameraConfiguration *PipelineHandlerRPi::generateConfiguration(Camera *camera,
>  
>  		case StreamRole::VideoRecording:
>  			fmts = data->isp_[Isp::Output0].dev()->formats();
> -			pixelFormat = formats::NV12;
> +			pixelFormat = formats::YUV420;

Does this mean that the colour denoise can only run when outputting
YUV420 ? Will it be silently disabled otherwise ? You set below the same
pixel format for outputs 0 and 1, if colour denoise can only run when
the format is YUV420, should we skip enabling output 1 if a different
pixel format has been selected, as that would waste resources ?

Actually, now that I think about it, and unless I'm mistaken, the
pipeline handler doesn't allocate and queue buffers for output 1 when
only output 0 is used by the application, so maybe it doesn't waste
resources and it's fine as is ?

>  			size = { 1920, 1080 };
>  			bufferCount = 4;
>  			outCount++;
> @@ -608,6 +608,7 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config)
>  	 * StreamConfiguration appropriately.
>  	 */
>  	V4L2DeviceFormat format;
> +	bool output1Set = false;
>  	for (unsigned i = 0; i < config->size(); i++) {
>  		StreamConfiguration &cfg = config->at(i);
>  
> @@ -632,6 +633,9 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config)
>  		format.size = cfg.size;
>  		format.fourcc = fourcc;
>  
> +		LOG(RPI, Info) << "Setting " << stream->name() << " to a resolution of "
> +			       << format.toString();

Should this be a debug message ? I'd also write

		LOG(RPI, Debug) << "Setting " << stream->name() << " to "
				<< format.toString();

as format.toString() will output both the resolution and pixel format.

> +
>  		ret = stream->dev()->setFormat(&format);
>  		if (ret)
>  			return -EINVAL;
> @@ -645,6 +649,35 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config)
>  
>  		cfg.setStream(stream);
>  		stream->setExternal(true);
> +
> +		if (i != maxIndex)
> +			output1Set = true;
> +	}
> +
> +	/*
> +	 * If ISP::Output1 stream has not been requested by the application, we
> +	 * set it up for internal use now. This second stream will be used for
> +	 * fast colour denoise, and must be a quarter resolution of the ISP::Output0
> +	 * stream. However, also limit the maximum size to 1200 pixels in the
> +	 * larger dimension, just to avoid being wasteful with buffer allocations
> +	 * and memory bandwidth.
> +	 */
> +	if (!output1Set) {
> +		V4L2DeviceFormat output1Format = format;
> +		constexpr unsigned int maxDimensions = 1200;
> +		const Size limit = Size(maxDimensions, maxDimensions).boundedToAspectRatio(format.size);

You could also write

		constexpr Size maxDimensions(1200, 1200);
		const Size limit = maxDimensions.boundedToAspectRatio(format.size);

Up to you.

> +
> +		output1Format.size = (format.size / 2).boundedTo(limit).alignedDownTo(2, 2);
> +
> +		LOG(RPI, Info) << "Setting ISP Output1 (internal) to a resolution of "
> +			       << output1Format.toString();

Same comment as above.

> +
> +		ret = data->isp_[Isp::Output1].dev()->setFormat(&output1Format);
> +		if (ret) {
> +			LOG(RPI, Error) << "Failed to set format on ISP Output1: "
> +					<< ret;
> +			return -EINVAL;
> +		}
>  	}
>  
>  	/* ISP statistics output format. */
Naushir Patuck Feb. 5, 2021, 1:38 p.m. UTC | #2 
Hi Laurent,

Thank you for your review feedback.

On Thu, 4 Feb 2021 at 21:32, Laurent Pinchart <
laurent.pinchart@ideasonboard.com> wrote:

> Hi Naush,
>
> Thank you for the patch.
>
> On Fri, Jan 29, 2021 at 03:11:50PM +0000, Naushir Patuck wrote:
> > In preparation for fast colour denoise, set the low resolution ISP
> > output stream (Output1) to a 1/4 resolution of the application requested
> > stream (Output0). This only happens if the application has not requested
> > an additional YUV or RGB stream.
> >
> > We also constrain this 1/4 resolution to at most 1200 pixels in the
> > largest dimension to avoid being too taxing on memory usage and system
> > bandwidth.
> >
> > Also switch the default StreamRole::VideoRecording to YUV420 to allow
> > fast colour denoise to run.
> >
> > Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
> > Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
> > Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> > ---
> >  .../pipeline/raspberrypi/raspberrypi.cpp      | 35 ++++++++++++++++++-
> >  1 file changed, 34 insertions(+), 1 deletion(-)
> >
> > diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
> b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
> > index d9895c779725..fe4c75f09925 100644
> > --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
> > +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
> > @@ -496,7 +496,7 @@ CameraConfiguration
> *PipelineHandlerRPi::generateConfiguration(Camera *camera,
> >
> >               case StreamRole::VideoRecording:
> >                       fmts = data->isp_[Isp::Output0].dev()->formats();
> > -                     pixelFormat = formats::NV12;
> > +                     pixelFormat = formats::YUV420;
>
> Does this mean that the colour denoise can only run when outputting
> YUV420 ? Will it be silently disabled otherwise ? You set below the same
> pixel format for outputs 0 and 1, if colour denoise can only run when
> the format is YUV420, should we skip enabling output 1 if a different
> pixel format has been selected, as that would waste resources ?
>

The colour denoise will only run if both image formats are YUV420.
Otherwise it will silently fail.  There is an optimisation to be had where
we could disable the output 1 stream if the format is not YUV420 and the
application has not requested a second stream, however see below....


>
> Actually, now that I think about it, and unless I'm mistaken, the
> pipeline handler doesn't allocate and queue buffers for output 1 when
> only output 0 is used by the application, so maybe it doesn't waste
> resources and it's fine as is ?
>

Sadly, this channel is currently always enabled - even if never used.  I
need to do more rigorous testing of our drivers to ensure there is no
problem with disabling only output 1.  Given that it would be default
behavior, I would like some confidence on the change before submitting.
This is also somewhat related to some work that I recently touched upon for
disable Unicam embedded streams when a sensor does not use time.  I will
add a \todo here to remind me to get to it.


>
> >                       size = { 1920, 1080 };
> >                       bufferCount = 4;
> >                       outCount++;
> > @@ -608,6 +608,7 @@ int PipelineHandlerRPi::configure(Camera *camera,
> CameraConfiguration *config)
> >        * StreamConfiguration appropriately.
> >        */
> >       V4L2DeviceFormat format;
> > +     bool output1Set = false;
> >       for (unsigned i = 0; i < config->size(); i++) {
> >               StreamConfiguration &cfg = config->at(i);
> >
> > @@ -632,6 +633,9 @@ int PipelineHandlerRPi::configure(Camera *camera,
> CameraConfiguration *config)
> >               format.size = cfg.size;
> >               format.fourcc = fourcc;
> >
> > +             LOG(RPI, Info) << "Setting " << stream->name() << " to a
> resolution of "
> > +                            << format.toString();
>
> Should this be a debug message ? I'd also write
>
>                 LOG(RPI, Debug) << "Setting " << stream->name() << " to "
>                                 << format.toString();
>
> as format.toString() will output both the resolution and pixel format.
>

Ack.


>
> > +
> >               ret = stream->dev()->setFormat(&format);
> >               if (ret)
> >                       return -EINVAL;
> > @@ -645,6 +649,35 @@ int PipelineHandlerRPi::configure(Camera *camera,
> CameraConfiguration *config)
> >
> >               cfg.setStream(stream);
> >               stream->setExternal(true);
> > +
> > +             if (i != maxIndex)
> > +                     output1Set = true;
> > +     }
> > +
> > +     /*
> > +      * If ISP::Output1 stream has not been requested by the
> application, we
> > +      * set it up for internal use now. This second stream will be used
> for
> > +      * fast colour denoise, and must be a quarter resolution of the
> ISP::Output0
> > +      * stream. However, also limit the maximum size to 1200 pixels in
> the
> > +      * larger dimension, just to avoid being wasteful with buffer
> allocations
> > +      * and memory bandwidth.
> > +      */
> > +     if (!output1Set) {
> > +             V4L2DeviceFormat output1Format = format;
> > +             constexpr unsigned int maxDimensions = 1200;
> > +             const Size limit = Size(maxDimensions,
> maxDimensions).boundedToAspectRatio(format.size);
>
> You could also write
>
>                 constexpr Size maxDimensions(1200, 1200);
>                 const Size limit =
> maxDimensions.boundedToAspectRatio(format.size);
>
> Up to you.
>

Ack.


>
> > +
> > +             output1Format.size = (format.size /
> 2).boundedTo(limit).alignedDownTo(2, 2);
> > +
> > +             LOG(RPI, Info) << "Setting ISP Output1 (internal) to a
> resolution of "
> > +                            << output1Format.toString();
>
> Same comment as above.
>

Ack.

Regards,
Naush



>
> > +
> > +             ret =
> data->isp_[Isp::Output1].dev()->setFormat(&output1Format);
> > +             if (ret) {
> > +                     LOG(RPI, Error) << "Failed to set format on ISP
> Output1: "
> > +                                     << ret;
> > +                     return -EINVAL;
> > +             }
> >       }
> >
> >       /* ISP statistics output format. */
>
> --
> Regards,
>
> Laurent Pinchart
>

Patch
diff mbox series 

diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
index d9895c779725..fe4c75f09925 100644
--- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
+++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp
@@ -496,7 +496,7 @@  CameraConfiguration *PipelineHandlerRPi::generateConfiguration(Camera *camera,
 
 		case StreamRole::VideoRecording:
 			fmts = data->isp_[Isp::Output0].dev()->formats();
-			pixelFormat = formats::NV12;
+			pixelFormat = formats::YUV420;
 			size = { 1920, 1080 };
 			bufferCount = 4;
 			outCount++;
@@ -608,6 +608,7 @@  int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config)
 	 * StreamConfiguration appropriately.
 	 */
 	V4L2DeviceFormat format;
+	bool output1Set = false;
 	for (unsigned i = 0; i < config->size(); i++) {
 		StreamConfiguration &cfg = config->at(i);
 
@@ -632,6 +633,9 @@  int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config)
 		format.size = cfg.size;
 		format.fourcc = fourcc;
 
+		LOG(RPI, Info) << "Setting " << stream->name() << " to a resolution of "
+			       << format.toString();
+
 		ret = stream->dev()->setFormat(&format);
 		if (ret)
 			return -EINVAL;
@@ -645,6 +649,35 @@  int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config)
 
 		cfg.setStream(stream);
 		stream->setExternal(true);
+
+		if (i != maxIndex)
+			output1Set = true;
+	}
+
+	/*
+	 * If ISP::Output1 stream has not been requested by the application, we
+	 * set it up for internal use now. This second stream will be used for
+	 * fast colour denoise, and must be a quarter resolution of the ISP::Output0
+	 * stream. However, also limit the maximum size to 1200 pixels in the
+	 * larger dimension, just to avoid being wasteful with buffer allocations
+	 * and memory bandwidth.
+	 */
+	if (!output1Set) {
+		V4L2DeviceFormat output1Format = format;
+		constexpr unsigned int maxDimensions = 1200;
+		const Size limit = Size(maxDimensions, maxDimensions).boundedToAspectRatio(format.size);
+
+		output1Format.size = (format.size / 2).boundedTo(limit).alignedDownTo(2, 2);
+
+		LOG(RPI, Info) << "Setting ISP Output1 (internal) to a resolution of "
+			       << output1Format.toString();
+
+		ret = data->isp_[Isp::Output1].dev()->setFormat(&output1Format);
+		if (ret) {
+			LOG(RPI, Error) << "Failed to set format on ISP Output1: "
+					<< ret;
+			return -EINVAL;
+		}
 	}
 
 	/* ISP statistics output format. */