Rpi 4 & 5 Libcamera, preview streaming and image capture
diff mbox series

Message ID 3cb201dab8e2$371eb550$a55c1ff0$@radicalsystems.co.za
State Not Applicable
Headers show
Series
  • Rpi 4 & 5 Libcamera, preview streaming and image capture
Related show

Commit Message

jan@radicalsystems.co.za June 7, 2024, 1:54 p.m. UTC 
Hi Laurent & Kieran

I still have not solved my issue and I'm trying to get some insight into the process of streaming to preview and then capturing an image at the same time.
I'm very unfamiliar with Licamera and I'm trying to understand how the whole process is put together.

So, what is working is the preview streaming in Android and running Google MLKIT analytics.
But as soon as the image capture instruction is sent, it falls over.
And this is the trace of what goes wrong.

1. Android initiate createCaptureRequest
2. generic_frame_buffer_allocator.cpp => PlatformFrameBufferAllocator::Private::allocate
	2.1 gralloc_open
	2.2 hardwareModule_->methods->open
3. camera3_hal.cpp => hal_dev_open
	3.1 CameraHalManager::instance()->open
4. camera_device.cpp => Camera::open
	4.1 camera_->acquire
5. camera.cpp => Camera::qcquire
	5.1 isAccessAllowed(Private::CameraAvailabe) is where it fails, because it assumes the camera is not open I guess

This is the stack trace:

06-04 13:46:54.783  3213  3296 D Camera2CaptureRequestBuilder: createCaptureRequest
06-04 13:46:54.795  3324  3534 E libcamera: ERROR Camera camera.cpp:675 Camera in Running state trying acquire() requiring state Available
06-04 13:46:54.795  3324  3534 E libcamera: ERROR HAL camera_device.cpp:390 '/base/axi/pcie@120000/rp1/i2c@88000/imx708@1a': Failed to acquire the camera
06-04 13:46:54.795  3324  3534 E libcamera: ERROR HAL camera3_hal.cpp:81 Failed to open camera module '0'
06-04 13:46:54.796  3324  3534 E libcamera: FATAL HAL generic_frame_buffer_allocator.cpp:106 gralloc_open() failed: -87
06-04 13:46:54.796  3324  3534 D libcamera: Backtrace:

Can you guys maybe enlighten me how this process is supposed to work when previewing a streaming and then capturing the image?
Because this seems to be the hiccup with this.
Are there two devices that must be opened or is the image capture done by the streaming instance?
I see /dev/media0 to 3 gets created for the camera.

I would really appreciate it if you can give me some pointers.
We need the get this hardware solution going.

Cheers
Jan


-----Original Message-----
From: Laurent Pinchart <laurent.pinchart@ideasonboard.com> 
Sent: Tuesday, June 4, 2024 10:14 PM
To: jan@radicalsystems.co.za
Cc: libcamera-devel@lists.libcamera.org
Subject: Re: Rpi 4 & 5 Libcamera & Mmap permission issue on Android 14

Hello Jan,

On Tue, Jun 04, 2024 at 12:26:20PM +0200, jan@radicalsystems.co.za wrote:
> Hi Libcamera Fundis!
> 
> Having a problem capturing an image on Android 14 (& 13).
> 
> Tombstone looks like this:
> 
> 06-03 15:13:03.880 308 578 I DMABUFHEAPS: Using : Non-legacy ION heaps
> 06-03 15:13:03.880 308 578 I [minigbm:gbm_mesa_internals.cpp(233)]: 
> Found GPU v3d
> 06-03 15:13:04.052 308 359 E libcamera: ERROR Buffer 
> mapped_framebuffer.cpp:230 Failed to mmap plane: Permission denied
> 
> .... lots of Permission denied!
> 
> 06-03 15:13:11.282 308 359 E libcamera: ERROR Buffer 
> mapped_framebuffer.cpp:230 Failed to mmap plane: Permission denied
> 06-03 15:13:12.008 308 579 E libcamera: ERROR Camera camera.cpp:675 
> Camera in Running state trying acquire() requiring state Available
> 06-03 15:13:12.008 308 579 E libcamera: ERROR HAL 
> camera_device.cpp:390 '/base/axi/pcie@120000/rp1/i2c@88000/imx708@1a': 
> Failed to acquire the camera
> 06-03 15:13:12.008 308 579 E libcamera: ERROR HAL camera3_hal.cpp:81 Failed to open camera module '0'
> 06-03 15:13:12.008 308 579 E libcamera: FATAL HAL 
> generic_frame_buffer_allocator.cpp:106 gralloc_open() failed: -87
> 
> MLKIT Streaming & analyzing works until trying to capture an image the 
> above happens.
> 
> The various Android versions for Rpi5 I could find gave the same issue.
> 
> I tried Emteria, AOSP & LineageOS 13 & 14
> 
> Android built from Raspberry Vanilla and even replacing with later 
> libcamera had the same issue.
> 
> I’m not sure if this is purely a configuration problem, wrt to 
> policies or permissions or specifically to the RPi’s architecture.
>
> Please help, I can rebuild and test.
> 
> I have tried the latest code pulled into RaspVanilla, but had the same issue.

This doesn't ring a bell, but let's see if we can debug it.

The first step is to figure out if the mmap() parameters are right. In MappedFrameBuffer::MappedFrameBuffer(), at the location of the failure, could you print the value of fd, info.mapLength, and mmapFlags ?
Something like the following should do:


(I haven't tested it though)

--
Regards,

Laurent Pinchart

Comments

Mattijs Korpershoek July 12, 2024, 9:29 a.m. UTC | #1 
Hi Jan,

On ven., juin 07, 2024 at 15:54, <jan@radicalsystems.co.za> wrote:

> Hi Laurent & Kieran
>
> I still have not solved my issue and I'm trying to get some insight into the process of streaming to preview and then capturing an image at the same time.
> I'm very unfamiliar with Licamera and I'm trying to understand how the whole process is put together.
>
> So, what is working is the preview streaming in Android and running Google MLKIT analytics.
> But as soon as the image capture instruction is sent, it falls over.
> And this is the trace of what goes wrong.
>
> 1. Android initiate createCaptureRequest
> 2. generic_frame_buffer_allocator.cpp => PlatformFrameBufferAllocator::Private::allocate
> 	2.1 gralloc_open
> 	2.2 hardwareModule_->methods->open
> 3. camera3_hal.cpp => hal_dev_open
> 	3.1 CameraHalManager::instance()->open
> 4. camera_device.cpp => Camera::open
> 	4.1 camera_->acquire
> 5. camera.cpp => Camera::qcquire
> 	5.1 isAccessAllowed(Private::CameraAvailabe) is where it fails, because it assumes the camera is not open I guess
>
> This is the stack trace:
>
> 06-04 13:46:54.783  3213  3296 D Camera2CaptureRequestBuilder: createCaptureRequest
> 06-04 13:46:54.795  3324  3534 E libcamera: ERROR Camera camera.cpp:675 Camera in Running state trying acquire() requiring state Available
> 06-04 13:46:54.795  3324  3534 E libcamera: ERROR HAL camera_device.cpp:390 '/base/axi/pcie@120000/rp1/i2c@88000/imx708@1a': Failed to acquire the camera
> 06-04 13:46:54.795  3324  3534 E libcamera: ERROR HAL camera3_hal.cpp:81 Failed to open camera module '0'
> 06-04 13:46:54.796  3324  3534 E libcamera: FATAL HAL generic_frame_buffer_allocator.cpp:106 gralloc_open() failed: -87

This reminds of a bug I solved long ago:

Here, libcamera tries to open the camera HAL instead of the gralloc HAL.

I send a patch for this here:
https://patchwork.libcamera.org/patch/18655/

Can you confirm the above patch is in your tree?

It should be, since it's been merged a year ago.


> 06-04 13:46:54.796  3324  3534 D libcamera: Backtrace:
>
> Can you guys maybe enlighten me how this process is supposed to work when previewing a streaming and then capturing the image?
> Because this seems to be the hiccup with this.
> Are there two devices that must be opened or is the image capture done by the streaming instance?
> I see /dev/media0 to 3 gets created for the camera.
>
> I would really appreciate it if you can give me some pointers.
> We need the get this hardware solution going.
>
> Cheers
> Jan
>
>
> -----Original Message-----
> From: Laurent Pinchart <laurent.pinchart@ideasonboard.com> 
> Sent: Tuesday, June 4, 2024 10:14 PM
> To: jan@radicalsystems.co.za
> Cc: libcamera-devel@lists.libcamera.org
> Subject: Re: Rpi 4 & 5 Libcamera & Mmap permission issue on Android 14
>
> Hello Jan,
>
> On Tue, Jun 04, 2024 at 12:26:20PM +0200, jan@radicalsystems.co.za wrote:
>> Hi Libcamera Fundis!
>> 
>> Having a problem capturing an image on Android 14 (& 13).
>> 
>> Tombstone looks like this:
>> 
>> 06-03 15:13:03.880 308 578 I DMABUFHEAPS: Using : Non-legacy ION heaps
>> 06-03 15:13:03.880 308 578 I [minigbm:gbm_mesa_internals.cpp(233)]: 
>> Found GPU v3d
>> 06-03 15:13:04.052 308 359 E libcamera: ERROR Buffer 
>> mapped_framebuffer.cpp:230 Failed to mmap plane: Permission denied
>> 
>> .... lots of Permission denied!
>> 
>> 06-03 15:13:11.282 308 359 E libcamera: ERROR Buffer 
>> mapped_framebuffer.cpp:230 Failed to mmap plane: Permission denied
>> 06-03 15:13:12.008 308 579 E libcamera: ERROR Camera camera.cpp:675 
>> Camera in Running state trying acquire() requiring state Available
>> 06-03 15:13:12.008 308 579 E libcamera: ERROR HAL 
>> camera_device.cpp:390 '/base/axi/pcie@120000/rp1/i2c@88000/imx708@1a': 
>> Failed to acquire the camera
>> 06-03 15:13:12.008 308 579 E libcamera: ERROR HAL camera3_hal.cpp:81 Failed to open camera module '0'
>> 06-03 15:13:12.008 308 579 E libcamera: FATAL HAL 
>> generic_frame_buffer_allocator.cpp:106 gralloc_open() failed: -87
>> 
>> MLKIT Streaming & analyzing works until trying to capture an image the 
>> above happens.
>> 
>> The various Android versions for Rpi5 I could find gave the same issue.
>> 
>> I tried Emteria, AOSP & LineageOS 13 & 14
>> 
>> Android built from Raspberry Vanilla and even replacing with later 
>> libcamera had the same issue.
>> 
>> I’m not sure if this is purely a configuration problem, wrt to 
>> policies or permissions or specifically to the RPi’s architecture.
>>
>> Please help, I can rebuild and test.
>> 
>> I have tried the latest code pulled into RaspVanilla, but had the same issue.
>
> This doesn't ring a bell, but let's see if we can debug it.
>
> The first step is to figure out if the mmap() parameters are right. In MappedFrameBuffer::MappedFrameBuffer(), at the location of the failure, could you print the value of fd, info.mapLength, and mmapFlags ?
> Something like the following should do:
>
> diff --git a/src/libcamera/mapped_framebuffer.cpp b/src/libcamera/mapped_framebuffer.cpp
> index b3104e0504d4..baa0587b4fcf 100644
> --- a/src/libcamera/mapped_framebuffer.cpp
> +++ b/src/libcamera/mapped_framebuffer.cpp
> @@ -223,6 +223,9 @@ MappedFrameBuffer::MappedFrameBuffer(const FrameBuffer *buffer, MapFlags flags)
>  		const int fd = plane.fd.get();
>  		auto &info = mappedBuffers[fd];
>  		if (!info.address) {
> +			LOG(Buffer, Info)
> +				<< "Mapping fd: " << fd << ", length: "
> +				<< info.mapLength << ", flags: " << mmapFlags;
>  			void *address = mmap(nullptr, info.mapLength, mmapFlags,
>  					     MAP_SHARED, fd, 0);
>  			if (address == MAP_FAILED) {
>
> (I haven't tested it though)
>
> --
> Regards,
>
> Laurent Pinchart
jan@radicalsystems.co.za July 12, 2024, 10:11 a.m. UTC | #2 
Hi Mattjis

Thank for reaching out 😊

No, this is still and ongoing saga.

I think that patch was somewhere and got lost again.
But yes, I did test that.
I got that fixed by moving gralloc to android libui, which adds a layer on top of that with extra checking.

My suspicion is the problem came in with Android 13 & 14.

Libui code looks very similar:
https://android.googlesource.com/platform/frameworks/native/+/a5529c8778c2f407f482fc12165aeb76c0f505c2/libs/ui/GraphicBufferAllocator.cpp

Here I have documented the issue
https://github.com/RadicalES/android_external_libcamera/issues/1

And solution:
https://github.com/RadicalES/android_external_libcamera/commit/b9aff6d3372e9702d646ec2603a90e28cef4c0c9

My next issue is a mmap permission problem detailed here:
https://github.com/RadicalES/android_external_libcamera/issues/2

I have tested code patched from the raspberry pi libcamera repo, Android Vanilla Repo and main libcamera repo.
That’s why ended up with my own repo trying to track the changes.

Once again thank you for the response!

Cheers
Jan
> -----Original Message-----
> From: Mattijs Korpershoek <mkorpershoek@baylibre.com>
> Sent: Friday, July 12, 2024 11:30 AM
> To: jan@radicalsystems.co.za; 'Laurent Pinchart'
> <laurent.pinchart@ideasonboard.com>
> Cc: libcamera-devel@lists.libcamera.org
> Subject: Re: Rpi 4 & 5 Libcamera, preview streaming and image capture
> 
> Hi Jan,
> 
> On ven., juin 07, 2024 at 15:54, <jan@radicalsystems.co.za> wrote:
> 
> > Hi Laurent & Kieran
> >
> > I still have not solved my issue and I'm trying to get some insight into the
> process of streaming to preview and then capturing an image at the same
> time.
> > I'm very unfamiliar with Licamera and I'm trying to understand how the
> whole process is put together.
> >
> > So, what is working is the preview streaming in Android and running Google
> MLKIT analytics.
> > But as soon as the image capture instruction is sent, it falls over.
> > And this is the trace of what goes wrong.
> >
> > 1. Android initiate createCaptureRequest 2.
> > generic_frame_buffer_allocator.cpp =>
> PlatformFrameBufferAllocator::Private::allocate
> > 	2.1 gralloc_open
> > 	2.2 hardwareModule_->methods->open
> > 3. camera3_hal.cpp => hal_dev_open
> > 	3.1 CameraHalManager::instance()->open
> > 4. camera_device.cpp => Camera::open
> > 	4.1 camera_->acquire
> > 5. camera.cpp => Camera::qcquire
> > 	5.1 isAccessAllowed(Private::CameraAvailabe) is where it fails,
> > because it assumes the camera is not open I guess
> >
> > This is the stack trace:
> >
> > 06-04 13:46:54.783  3213  3296 D Camera2CaptureRequestBuilder:
> > createCaptureRequest
> > 06-04 13:46:54.795  3324  3534 E libcamera: ERROR Camera
> > camera.cpp:675 Camera in Running state trying acquire() requiring
> > state Available
> > 06-04 13:46:54.795  3324  3534 E libcamera: ERROR HAL
> > camera_device.cpp:390
> '/base/axi/pcie@120000/rp1/i2c@88000/imx708@1a':
> > Failed to acquire the camera
> > 06-04 13:46:54.795  3324  3534 E libcamera: ERROR HAL camera3_hal.cpp:81
> Failed to open camera module '0'
> > 06-04 13:46:54.796  3324  3534 E libcamera: FATAL HAL
> > generic_frame_buffer_allocator.cpp:106 gralloc_open() failed: -87
> 
> This reminds of a bug I solved long ago:
> 
> Here, libcamera tries to open the camera HAL instead of the gralloc HAL.
> 
> I send a patch for this here:
> https://patchwork.libcamera.org/patch/18655/
> 
> Can you confirm the above patch is in your tree?
> 
> It should be, since it's been merged a year ago.
> 
> 
> > 06-04 13:46:54.796  3324  3534 D libcamera: Backtrace:
> >
> > Can you guys maybe enlighten me how this process is supposed to work
> when previewing a streaming and then capturing the image?
> > Because this seems to be the hiccup with this.
> > Are there two devices that must be opened or is the image capture done
> by the streaming instance?
> > I see /dev/media0 to 3 gets created for the camera.
> >
> > I would really appreciate it if you can give me some pointers.
> > We need the get this hardware solution going.
> >
> > Cheers
> > Jan
> >
> >
> > -----Original Message-----
> > From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > Sent: Tuesday, June 4, 2024 10:14 PM
> > To: jan@radicalsystems.co.za
> > Cc: libcamera-devel@lists.libcamera.org
> > Subject: Re: Rpi 4 & 5 Libcamera & Mmap permission issue on Android 14
> >
> > Hello Jan,
> >
> > On Tue, Jun 04, 2024 at 12:26:20PM +0200, jan@radicalsystems.co.za wrote:
> >> Hi Libcamera Fundis!
> >>
> >> Having a problem capturing an image on Android 14 (& 13).
> >>
> >> Tombstone looks like this:
> >>
> >> 06-03 15:13:03.880 308 578 I DMABUFHEAPS: Using : Non-legacy ION
> >> heaps
> >> 06-03 15:13:03.880 308 578 I [minigbm:gbm_mesa_internals.cpp(233)]:
> >> Found GPU v3d
> >> 06-03 15:13:04.052 308 359 E libcamera: ERROR Buffer
> >> mapped_framebuffer.cpp:230 Failed to mmap plane: Permission denied
> >>
> >> .... lots of Permission denied!
> >>
> >> 06-03 15:13:11.282 308 359 E libcamera: ERROR Buffer
> >> mapped_framebuffer.cpp:230 Failed to mmap plane: Permission denied
> >> 06-03 15:13:12.008 308 579 E libcamera: ERROR Camera camera.cpp:675
> >> Camera in Running state trying acquire() requiring state Available
> >> 06-03 15:13:12.008 308 579 E libcamera: ERROR HAL
> >> camera_device.cpp:390
> '/base/axi/pcie@120000/rp1/i2c@88000/imx708@1a':
> >> Failed to acquire the camera
> >> 06-03 15:13:12.008 308 579 E libcamera: ERROR HAL camera3_hal.cpp:81
> Failed to open camera module '0'
> >> 06-03 15:13:12.008 308 579 E libcamera: FATAL HAL
> >> generic_frame_buffer_allocator.cpp:106 gralloc_open() failed: -87
> >>
> >> MLKIT Streaming & analyzing works until trying to capture an image
> >> the above happens.
> >>
> >> The various Android versions for Rpi5 I could find gave the same issue.
> >>
> >> I tried Emteria, AOSP & LineageOS 13 & 14
> >>
> >> Android built from Raspberry Vanilla and even replacing with later
> >> libcamera had the same issue.
> >>
> >> I’m not sure if this is purely a configuration problem, wrt to
> >> policies or permissions or specifically to the RPi’s architecture.
> >>
> >> Please help, I can rebuild and test.
> >>
> >> I have tried the latest code pulled into RaspVanilla, but had the same
> issue.
> >
> > This doesn't ring a bell, but let's see if we can debug it.
> >
> > The first step is to figure out if the mmap() parameters are right. In
> MappedFrameBuffer::MappedFrameBuffer(), at the location of the failure,
> could you print the value of fd, info.mapLength, and mmapFlags ?
> > Something like the following should do:
> >
> > diff --git a/src/libcamera/mapped_framebuffer.cpp
> > b/src/libcamera/mapped_framebuffer.cpp
> > index b3104e0504d4..baa0587b4fcf 100644
> > --- a/src/libcamera/mapped_framebuffer.cpp
> > +++ b/src/libcamera/mapped_framebuffer.cpp
> > @@ -223,6 +223,9 @@ MappedFrameBuffer::MappedFrameBuffer(const
> FrameBuffer *buffer, MapFlags flags)
> >  		const int fd = plane.fd.get();
> >  		auto &info = mappedBuffers[fd];
> >  		if (!info.address) {
> > +			LOG(Buffer, Info)
> > +				<< "Mapping fd: " << fd << ", length: "
> > +				<< info.mapLength << ", flags: " <<
> mmapFlags;
> >  			void *address = mmap(nullptr, info.mapLength,
> mmapFlags,
> >  					     MAP_SHARED, fd, 0);
> >  			if (address == MAP_FAILED) {
> >
> > (I haven't tested it though)
> >
> > --
> > Regards,
> >
> > Laurent Pinchart

Patch
diff mbox series 

diff --git a/src/libcamera/mapped_framebuffer.cpp b/src/libcamera/mapped_framebuffer.cpp
index b3104e0504d4..baa0587b4fcf 100644
--- a/src/libcamera/mapped_framebuffer.cpp
+++ b/src/libcamera/mapped_framebuffer.cpp
@@ -223,6 +223,9 @@  MappedFrameBuffer::MappedFrameBuffer(const FrameBuffer *buffer, MapFlags flags)
 		const int fd = plane.fd.get();
 		auto &info = mappedBuffers[fd];
 		if (!info.address) {
+			LOG(Buffer, Info)
+				<< "Mapping fd: " << fd << ", length: "
+				<< info.mapLength << ", flags: " << mmapFlags;
 			void *address = mmap(nullptr, info.mapLength, mmapFlags,
 					     MAP_SHARED, fd, 0);
 			if (address == MAP_FAILED) {