[camera] Camera FPS of recorded video and the preview is really low ~18fps

VERIFIED FIXED in Firefox 22

Status

--
major
VERIFIED FIXED
6 years ago
6 years ago

People

(Reporter: ikumar, Assigned: sotaro)

Tracking

unspecified
B2G C4 (2jan on)
ARM
Gonk (Firefox OS)

Firefox Tracking Flags

(blocking-b2g:tef+, firefox20 wontfix, firefox21 wontfix, firefox22 fixed, b2g18 fixed, b2g18-v1.0.0 wontfix, b2g18-v1.0.1 fixed)

Details

Attachments

(4 attachments)

(Reporter)

Description

6 years ago
I have been investigating why the fps of the recorded videos and preview are really low (~18fps) on b2g and using the same hardware on android it's around 29 fps.
After working with camera team it turns out that the dequeuebuffer() in GonkNativeWindow.cpp is taking long time on b2g (> 50ms). I instrumented the code and confirm that most of the time it's > 50ms which causing the low FPS.

We need to optimize this function to get better FPS.
(Reporter)

Comment 1

6 years ago
Created attachment 722412 [details]
DequeueBuffer performance on FFOS
(Reporter)

Comment 2

6 years ago
Created attachment 722415 [details]
DequeueBuffer performance on Android

You can see how it takes so much more time on FFOS. On Android it max goes to the order of 15ms where as on FFOS it takes up to .7 seconds!
The numbers were collected on QRD.
(Reporter)

Updated

6 years ago
blocking-b2g: --- → tef?
blocking-b2g: tef? → tef+
Hi Ben,
Please help with this one.
thanks.
Assignee: nobody → btian
dequeueBuffer() takes as long as it does because it often blocks waiting for a buffer to get returned from the viewfinder MediaStream.  (I'm not certain why this takes so long, but it may be because the MediaStream likes to keep extra frames in its queue, depleting the pipeline.)

I've CCed sotaro, since he was working on bug 844248 to address this issue.

Updated

6 years ago
Depends on: 844248

Comment 5

6 years ago
Can we unhide this bug?
Group: qualcomm-confidential
(Assignee)

Comment 6

6 years ago
(In reply to Inder from comment #0)
> I have been investigating why the fps of the recorded videos and preview are
> really low (~18fps) on b2g and using the same hardware on android it's
> around 29 fps.
> After working with camera team it turns out that the dequeuebuffer() in
> GonkNativeWindow.cpp is taking long time on b2g (> 50ms). I instrumented the
> code and confirm that most of the time it's > 50ms which causing the low FPS.

The problem happens because GonkNativeWindow throws gralloc buffers out of the class and the buffers are rendered in Compositor. Camera hal tries to dequeue buffers from GonkNativeWindow until a number of UNDEQUEUED_BUFFERS becomes MIN_UNDEQUEUED_BUFFERS(2). But more than 2 buffers are out of GonkNativeWindow. Then Camera hal always needs to wait buffer return.

There are 2 ways to solve this problem. Both ways are very risky and need more gralloc buffers.
[1] Increment MIN_UNDEQUEUED_BUFFERS from 2 to some value. Ensure the max number of gralloc buffers between GonkNativeWindow and Compositor. Modify camera hal to accept MIN_UNDEQUEUED_BUFFERS more than 2.

[2] Copy preview frames for rendering and GonkNativeWindow run in async mode.  The copied frames are throwed out of GonkNativeWindow for rendering. Original frames stay within GonkNativeWindow.

[1] needs nodification to camera hal and MediaStreamGraph. And more GraphicBuffers
Camera hal of unagi does not work in MIN_UNDEQUEUED_BUFFERS more than 2.

[2] needs a cpu time for copy video frames. And more GraphicBuffers. But it does not scale to high resolution preview.
(Assignee)

Comment 7

6 years ago
Created attachment 724576 [details]
Crash log of unagi when UNDEQUEUED_BUFFERS is 3

When I change the UNDEQUEUED_BUFFERS from 2 to 3 on unagi. The camera hal cause abort with the log during starting preview.
Inder, what do you think about ^^^?
Flags: needinfo?(ikumar)
(Assignee)

Comment 9

6 years ago
(In reply to Sotaro Ikeda [:sotaro] from comment #6)
> (In reply to Inder from comment #0)
//snip
> There are 2 ways to solve this problem. Both ways are very risky and need
> more gralloc buffers.
> [1] Increment MIN_UNDEQUEUED_BUFFERS from 2 to some value. Ensure the max
> number of gralloc buffers between GonkNativeWindow and Compositor. Modify
> camera hal to accept MIN_UNDEQUEUED_BUFFERS more than 2.
> 
> [2] Copy preview frames for rendering and GonkNativeWindow run in async
> mode.  The copied frames are throwed out of GonkNativeWindow for rendering.
> Original frames stay within GonkNativeWindow.

In Bug 844248, I created attachment 724585 [details] [diff] [review] as temporary patch of [2]. Copying of video frame(480*320 in camera mode) takes 10ms-25ms(huge!). And after when I took a picture, camera app abort because of out of memory(gralloc buffer) during restarting preview.

Then [1] is a correct way to fix the problem.
(Reporter)

Comment 10

6 years ago
(In reply to Michael Vines [:m1] [:evilmachines] from comment #8)
> Inder, what do you think about ^^^?

Sotaro, I am still not clear about why Android works well with MIN_UNDEQUEUED_BUFFERS=2 whereas on FFOS we need >2.
Flags: needinfo?(ikumar)
Can we bypass MediaStreamGraph to show camera preview on display directly? I wonder the necessity of MediaStreamGraph for camera preview and didn't get why we need it to render camera preview to video tag.
(Assignee)

Comment 12

6 years ago
(In reply to Inder from comment #10)
> (In reply to Michael Vines [:m1] [:evilmachines] from comment #8)
> > Inder, what do you think about ^^^?
> 
> Sotaro, I am still not clear about why Android works well with
> MIN_UNDEQUEUED_BUFFERS=2 whereas on FFOS we need >2.

<SurfaceTexture in android case>
Buffers are managed by following states in SurfaceTexture
 - FREE, DEQUEUED, QUEUED
And when rendering camera preview, SurfaceTexture works in async mode.
In the async mode, only one buffer is set to QUEUED state.
And currentTexture's buffer is set to FREE state.
The above two buffers are need to be undequeued. 
So, MIN_UNDEQUEUED_BUFFERS=2 works.

<GonkNativeWindow in Firefox OS case>
Buffers are managed by following states in SurfaceTexture
 - FREE, DEQUEUED, QUEUED, RENDERING

Buffers are set to RENDERING state when buffers are sent to the compositor.
The number of buffers in RENDERING state exceed more than two.
MediaStreamGraph is one big area of buffering buffers.
Then MIN_UNDEQUEUED_BUFFERS=2 do not works.
(Assignee)

Comment 13

6 years ago
Also in android, SurfaceMediaSource(ISurfaceTexture derived class) sets MIN_UNDEQUEUED_BUFFERS=4.

http://androidxref.com/4.0.4/xref/frameworks/base/include/media/stagefright/SurfaceMediaSource.h#37
(Assignee)

Comment 14

6 years ago
(In reply to Ben Tian [:bentian] from comment #11)
> Can we bypass MediaStreamGraph to show camera preview on display directly? I
> wonder the necessity of MediaStreamGraph for camera preview and didn't get
> why we need it to render camera preview to video tag.

I do not understand well about "show camera preview on display directly?". Can you explain about it? Camera preview needs to be layouted and buffers of the preview frame needs to be sent to compositor via ImageContainerChild. Followings is a diagram of current configuration.

https://github.com/sotaroikeda/firefox-diagrams/blob/master/dom/dom_camera_FirefoxOS_1_1.pdf?raw=true
(In reply to Sotaro Ikeda [:sotaro] from comment #14)
> 
> I do not understand well about "show camera preview on display directly?".

Instead of setting the mozSrcObject of a <video> tag to a MediaStream, could we just have an <img> tag and somehow push frames to it?
(Assignee)

Comment 16

6 years ago
camera needs to provide MediaStream also for other use like Bug 842243. I am thinking to create custom MediaStream for camera preview to solve buffering and delay problem.
(In reply to Sotaro Ikeda [:sotaro] from comment #14)
> I do not understand well about "show camera preview on display directly?".
> Can you explain about it? Camera preview needs to be layouted and buffers of
> the preview frame needs to be sent to compositor via ImageContainerChild.
> Followings is a diagram of current configuration.
> 
> https://github.com/sotaroikeda/firefox-diagrams/blob/master/dom/
> dom_camera_FirefoxOS_1_1.pdf?raw=true

I mean 'creating a shortcut from camera frame to ImageContainerChild.' My idea is to omit the whole MediaStreamGraph if camera preview doesn't require it.

Thanks for the camera diagrams. They help a lot.

(In reply to Sotaro Ikeda [:sotaro] from comment #16)
> camera needs to provide MediaStream also for other use like Bug 842243. I am
> thinking to create custom MediaStream for camera preview to solve buffering
> and delay problem.

This is another approach. I wonder whether 1) to omit MediaStreamGraph or 2) to customize it is less risky.
(Assignee)

Comment 18

6 years ago
In Bug 844248, I created a temporary patch for b2g18 attachment 726242 [details] [diff] [review] . 

By applying the patch, fps of preview becomes better and camera/video mode change becomes quicker.
:sotaro, since you are working on this, mind taking this bug?
(Assignee)

Comment 20

6 years ago
(In reply to Joe Cheng [:jcheng] from comment #19)
> :sotaro, since you are working on this, mind taking this bug?

jcheng, which problem are you going to solve?

From my point of view, the problem is divided to two problems.
[1] latency and buffering at MediaStreamGraph
[2] MIN_UNDEQUEUED_BUFFERS=2 is not enough for FirefoxOS 

I am going to fix [1] at Bug 844248. I got positive feedback about attachment 726207 [details] [diff] [review] from roc. I am going to fix [1] soon.

[2] needs partner's fix. I do not worikng about [2].
(Assignee)

Comment 21

6 years ago
(In reply to Sotaro Ikeda [:sotaro] from comment #20)
> (In reply to Joe Cheng [:jcheng] from comment #19)
> > :sotaro, since you are working on this, mind taking this bug?
> 
> jcheng, which problem are you going to solve?
> 
> From my point of view, the problem is divided to two problems.
> [1] latency and buffering at MediaStreamGraph
> [2] MIN_UNDEQUEUED_BUFFERS=2 is not enough for FirefoxOS 
> 
> I am going to fix [1] at Bug 844248. I got positive feedback about
> attachment 726207 [details] [diff] [review] from roc. I am going to fix [1]
> soon.

I am working for Bug 844248. So I do not mind you take this bug.
(Assignee)

Comment 22

6 years ago
(In reply to Joe Cheng [:jcheng] from comment #19)
> :sotaro, since you are working on this, mind taking this bug?

Sorry, I misunderstand about it. I am going to take the bug.
(Assignee)

Updated

6 years ago
Assignee: btian → sotaro.ikeda.g
(Assignee)

Comment 23

6 years ago
> > 
> > From my point of view, the problem is divided to two problems.
> > [1] latency and buffering at MediaStreamGraph
> > [2] MIN_UNDEQUEUED_BUFFERS=2 is not enough for FirefoxOS 
> > 

Inder, [1] is going to be fixed at Bug 844248 soon. This fix improve preview and recording quality very much.

Can you re-evaluate the quality again after the fix? It is enough for v1.01, I think.
(Reporter)

Comment 24

6 years ago
Sure, let me apply the patch and see how much improvement we get.
(Reporter)

Comment 25

6 years ago
Sotaro: with the change in bug 844248 I am seeing fps as ~29.5 which is same as what we get on Android! So, it definitely fixes this issue. Lets get the fix landed soon.
(Reporter)

Comment 26

6 years ago
Created attachment 727468 [details]
DequeueBuffer performance with Sotaro patch on FFOS

Just to make sure that the patch in bug 844248 is helping with the FPS performance. I measured the DequeueBuffer timing on FFOS with Sotaro's patch in bug 844248. There is tremendous improvement in the timing. Though there are instances when it does take > 20ms. But those instances are very few and doesn't affect the FPS performance which is around 29.5.
Inder, are we concerned about [2] from comment 20? 
Or per your comment 26 we're OK to close off this bug as a DUP to bug 844248 and perhaps track [2] elsewhere as it shouldn't block the release? (correct me if I have the wrong idea).
Flags: needinfo?(ikumar)
(Reporter)

Comment 28

6 years ago
We are able to get ~29.5 FPS by just fixing bug 844248 so I don't think we need make [2] change in comment 20. I am marking this bug resolved.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Flags: needinfo?(ikumar)
Resolution: --- → FIXED
Calling this fixed by bug 844248 per comment 28.
status-b2g18: --- → fixed
status-b2g18-v1.0.0: --- → wontfix
status-b2g18-v1.0.1: --- → fixed
status-firefox20: --- → wontfix
status-firefox21: --- → wontfix
status-firefox22: --- → fixed
Target Milestone: --- → B2G C4 (2jan on)

Comment 30

6 years ago
Verified Fixed on Unagi. With Show Frames per second setting enabled, user created video indicated 29-35 FPS.
Unagi Build ID: 20130322070203
Kernel Date: Dec 5
Gecko: http://hg.mozilla.org/releases/mozilla-b2g18_v1_0_1/rev/4931ec89ebbe
Gaia: 85fd164691bb852f1cfaf82405df4380629ced6e
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.