1463904 - 1080p60 video playback freezes on Oculus Go even though audio continues to play

Status: RESOLVED WONTFIX

(Core :: Audio/Video: Playback, defect, P2)

Description

[Randall Barker [:rbarker]]

Playing streamed videos from twitch.com on the Oculus Go causes excessive CPU utilization which results in process starvation and frozen video playback. The device should have adequate hardware to handle 1080p video streams. The user experience on twitch is particularly poor because it is sent 1080p video that plays a few frames then freezes and never resumes. By manually dropping the video to 720p, the video will usually resume.

Comment 1

[David Bolter [:davidb] (NeedInfo me for attention)]

P1 for FXR

Comment 2

[Chris Peterson [:cpeterson]]

Jean-Yves, GeckoView video playback on the Oculus Go VR headset is really bad. How do you recommend we debug this? How can we determine whether GeckoView video playback is using hardware or software decoding?

Oculus says the Samsung Galaxy S7 (US edition only!) has same hardware as Oculus Go headset. Jim offered to test on his Galaxy S7 to see if he can reproduce the problem. If he can't, then it is probably an issue with Oculus Go software instead of a hardware issue.

Comment 3

[Jean-Yves Avenard [:jya]]

Change to P2, as P1 are for crashers.

You can tell if hardware acceleration is used via the media devtools extension, however, it's never been tested via android.

I'm not familiar with geckoview, nor how it would handle hardware surfaces as returned by the hardware decoder.
With support for HW surface, then it would be using the software decoder, and doing 1080p might be problematic on any mobiles...

If someone could attach a debugger, we could tell.

Comment 4

[Jean-Yves Avenard [:jya]]

if you have high CPU, it's likely the HW decoder isn't used.

Comment 5

[Jean-Yves Avenard [:jya]]

Using Firefox nightly or release I can play 1080p60 twitch live stream on my galaxy S8 without issue whatsoever.

1080p60 also plays just fine on the Sony Z3C which is rather ancient when it comes to android phone. Can't notice any frames drop.

So my guess is that GeckoView doesn't do something right... what are the differences rendering wise between geckoView and Firefox?

Comment 6

[Jean-Yves Avenard [:jya]]

GeckoView isn't my component, so maybe you guys have different rules

Comment 7

[Jean-Yves Avenard [:jya]]

John is the expert here, he starts in 5 weeks, but he may have some ideas today.

Comment 8

[Randall Barker [:rbarker]]

Some additional information. Top when streaming 1080p60 video on twitch:

8270 u0_a50   20   0  33% S    59 1791696K 350564K  fg org.mozilla.vrbrowser:tab
8114 u0_a50   10 -10  10% S   159 1978048K 232008K  ta org.mozilla.vrbrowser
8399 u0_a50   20   0   9% R    19 1275324K  51956K  fg org.mozilla.vrbrowser:media

Video is frozen, audio plays.

Top when streaming same video at 720p on twitch:

8270 u0_a50   20   0  35% S    58 1793232K 310256K  fg org.mozilla.vrbrowser:tab
8114 u0_a50   10 -10  19% R   158 1962960K 231840K  ta org.mozilla.vrbrowser
8399 u0_a50   20   0   6% S    19 1261820K  51916K  fg org.mozilla.vrbrowser:media

Video and audio play smoothly.

Additionally we have verified that video is using hardware decode.

Streaming a similar 1080p video on a Galaxy S7 is smooth and with out issue.

Comment 9

[Chris Peterson [:cpeterson]]

Changing bug summary to indicate this is a problem with 1080p60 playback and not related to CPU usage.

Clearing the needinfo for jchen because he and Randall confirmed that the Galaxy S7 can play 1080p60 smoothly.

Comment 10

[Chris Peterson [:cpeterson]]

jya, if you (or John) would like an Oculus Go for testing, just let me know. We can order one for you. Alternatively, you can buy one retail and expense it.

Comment 11

[Jean-Yves Avenard [:jya]]

How is Android webview doing on those?

Just a thought: Something we encountered often with the android HW decoder was the number of available HW surfaces to which the decoder can output its content.
The decoder would stop outputting until a new surface was available, and a surface can only become available once it's been rendered.

The number of available surface is low, sometimes as low as 4.
If the video queue is too long, or the compositor queue to many of those, or not release them, then it would appear that video is frozen as the decoder no longer output anything.

The size of the video queue is controlled by a pref, media.video-queue.hw-accel-size it should be no bigger than 3 (the default if the pref isn't set)
You can also play with media.video-queue.default-size Which comtrols the default queue size, maybe somehow with geckoview the values are messed up, it could explain what you're seeing.

Also, IIRC, there's a piece of code that defines the size of the video surface allocated, maybe it's not correctly set and why 720p plays, but not 1080p

That no video frames are displayed at all tells me that it's not a decoding speed, as otherwise we would see at least static frames here and there (keyframes). So first step is making sure the video frame queue is set to the right value. Playing with MOZ_LOG=MediaDecoder:5,MediaFormatReader:5 would likely give a good clue on what's going on.

I have no doubt John could determine what was going quickly here, he's a magician in troubleshooting those.

Sorry if I can't be of more help at this stage, maybe after the all hands.

Comment 12

[Randall Barker [:rbarker]]

I put some logging into mobile/android/geckoview/src/main/java/org/mozilla/gecko/media/LollipopAsyncCodec.java and it never seems to stall. In fact the logging between 1080p60 and 720p look the same. Only difference is 720p actually plays.

Comment 13

[John Lin [:jhlin][:jolin]]

Sorry for the late reply.

It's often hard to tell what went wrong w/o log. Randall, could you please capture and upload logcat dumps for 1080 video playback?

Also, logging around [1] might help narrowing down where to continue the investigation. If RemoteVideoDecoder::CallbacksSupport doesn't stall either, my next step would be turning on logging in VideoSink[2] to see whether output surfaces are sent to the compositor or not.

[1] https://searchfox.org/mozilla-central/source/dom/media/platforms/android/RemoteDataDecoder.cpp#150
[2] https://searchfox.org/mozilla-central/source/dom/media/mediasink/VideoSink.cpp#28

Comment 14

[Chris Peterson [:cpeterson]]

Jean-Yves, can we configure GeckoView's advertised media capabilities so web content will prefer 720p over 1080p streams on the Oculus Go? This would at least avoid the platform problem until we can diagnose it.

Comment 15

[Chris Peterson [:cpeterson]]

Assigning to Jean-Yves for now. He and Randall debugged this problem at last week's All Hands and Jean-Yves plans to order his own Oculus Go headset to continue debugging.

Comment 16

[Jean-Yves Avenard [:jya]]

still waiting on the Go, was supposed to have been shipped on monday.. no news still

Comment 17

[Andreas Bovens [:abovens]]

Have you received the Go, Jean-Yves? Any updates?

Comment 18

[Jean-Yves Avenard [:jya]]

I have.

John has a theory on what the problem is, and how we could reduce the problem.

It's an extensive work however in the Android compositor.. He's working on it.

Additionally, I have started to personally look into bug 1245400. While this won't fix the 1080p60 issue, it will allow sites like Twitch to properly detect that we're dropping frames and to switch to a lower resolution.
Which is what Twitch should have done all along.

Comment 19

[Chris Peterson [:cpeterson]]

This is an FxR bug. Marking as [geckoview:klar:p2] because this bug is not a Focus+GV blocker.

Comment 20

[John Lin [:jhlin][:jolin]]

Took me a while but now I have a better theory.

At first I thought it was because that on Android, we send only one frame to compositor at a time, which doesn't take advantage of what roc did in bug Bug 1143575. And to fix that, we have to find a way to specify timestamp by calling the API 21+ method MediaCodec.releaseOutputBuffer [1] and make it work with Gecko composite system. However, that didn't explain why this problem happens only on VR devices and not on phones.

So I did a series of tests and study the code of Firefox Reality to further investigate the problem. Long story short, IIUC, Firefox Reality uses a SurfaceTexture as the render target of GeckoView, and that texture is used when rendering VR scene. It's possible that the sync object [2] of the texture makes blocking happen more frequently and output buffers take more time to be released to the decoder.

I'll do some more tests and observe if the result can verify this theory. If so, then double buffering technique might be of some help here.

[1] https://developer.android.com/reference/android/media/MediaCodec.html#releaseOutputBuffer(int,%20long)
[2] https://android.googlesource.com/platform/frameworks/native/+/lollipop-release/libs/gui/GLConsumer.cpp#402

Comment 21

[John Lin [:jhlin][:jolin]]

My 2nd theory (about the SurfaceTexture) doesn't stand either. In fact, process starvation that Randall described in comment 0, is the issue here.

What I found so far:

1. Oculus Go
 - the baseline of available CPU time for apps is lower. While Oculus Go and Galaxy S7 have the same hardware, the software they run is quite different. For example, there are some RT (real-time) and high priority(with negative nice value) processes/threads run constantly on Oculus GO to track device position/motion. On phones, they only run in certain apps (usually games and AR/VR apps).
 - CPU governor doesn't work as hoped. According to [1], the governor clocks the CPU according to utilization and missed frames. In our case, the utilization is high but almost no missed frames (the frames were dropped in media stack, so compositor/render is unaware of them).

2. Twitch
 - the JS workload is higher. The streams on Twitch are originally HLS contents (m3u8+mpeg2-ts) and are transcoded into MSE (fragmented mp4) live.
 - no periodic keyframes for video to catch up. When video decoding is behind the (audio) playback position, our media stack tries to skip to the next keyframe, but Twitch streams don't serve periodic keyframes.

3. FirefoxReality
 - High priority threads preempt aggressively. Especially the looper thread (raised at [2]) and render thread.

5. Gecko media stack & Android codec
 - MediaCodec sometimes emits decoded frame out of order, which won't be dropped until VideoSink found them.
 - MediaFormatReader doesn't update playback position to platform decoder for output buffers to be recycled asap.
  
What to do (IMHO) to address the problem:

1. I don't think there is much we can do to increase CPU availability other than setting higher CPU clock level. However, to avoid heating and battery issues, we need to either implement our own dynamic throttler or expose the missed video frames to the compositor/renderer so that system CPU governor can work properly. The latter could be complicated because rendering is not just for video, but for other visible elements updates too.

2. Ask Twitch if they'd like to serve HLS contents to FirefoxReality (and other GeckoView based products). Streams with periodic keyframes help, too.

3. Lower the priorities of looper and renderer threads to minimal required level. It might impact the experience and needs careful tunning, I suppose.

4. Detect useless frames (out of order, behind playback position) and recycle them asap. This is beneficial to all GeckoView apps and I'll file other bugs to work on them.

[1] https://developer.oculus.com/blog/optimizing-oculus-go-for-performance/
[2] https://github.com/MozillaReality/FirefoxReality/blob/562515f1bc7e3a4f25c90d367c6b4ec8664711fa/app/src/oculusvr/cpp/DeviceDelegateOculusVR.cpp#L507

Comment 22

[Chris Peterson [:cpeterson]]

There's not much we can do without help from Oculus.

Comment 23

[Lars Bergstrom (:larsberg)]

John, are things any better now? Randall mentioned that twitch.tv is now automatically downgrading to 720p. Is there any progress or further ideas on things we could do here?

I'm also not clear - what are we asking Oculus to help with?

Comment 24

[John Lin [:jhlin][:jolin]]

(In reply to Lars Bergstrom (:larsberg) from comment #23)
> John, are things any better now? Randall mentioned that twitch.tv is now
> automatically downgrading to 720p. Is there any progress or further ideas on
> things we could do here?

Unfortunately, no. In my tests, in order to play 1080p@60fps Twitch streams on Oculus Go, the CPUs must be at high clock and video playback threads must have the same priority as the perf threads to prevent preempting. But it causes both thermal increase and battery drain. In this particular case, playing 720p or 30fps sounds better than burning headsets, IMHO.

> 
> I'm also not clear - what are we asking Oculus to help with?

I guess it might be helpful to inform Oculus about the CPU starvation situation for suggestions for better scheduling playback threads (other than making them all perf threads). Knowing why the governor doesn't clock up CPUs even when our threads are starving would be nice, too.


(Un-assign myself from this bug because there is no further action in media playback pipeline to address the problem for now.)

Comment 25

[Chris Peterson [:cpeterson]]

P2. There is not much we can do when the media process is starved. We should talk to Oculus.

In the meantime, can GeckoView report media capabilities (bug 1409664) that will cause web content to prefer 720p streams over 1080p? This would at least avoid the platform problem until we can diagnose it.

Comment 26

[Jean-Yves Avenard [:jya]]

(In reply to Chris Peterson [:cpeterson] from comment #25)
> P2. There is not much we can do when the media process is starved. We should
> talk to Oculus.
> 
> In the meantime, can GeckoView report media capabilities (bug 1409664) that
> will cause web content to prefer 720p streams over 1080p? This would at
> least avoid the platform problem until we can diagnose it.

Twitch should instead look at the dropped frames count and select 720p. It's the proper way to do it.

Comment 27

[Kelsey Gilbert [:jgilbert]]

It really sounds like we need to expose missed frames to the OS compositor. That's the mechanism the system is designed for, so anything else is going to be trickier and more fragile.

Traditionally, we try to keep our browser-chrome (ui) at full framerate, even if content falls behind. Particularly for VR, content framerate is equally (more?) important than ui framerate.

mattwoodrow:
Does that sound right? How crazy would it be to allow compositing to drop to (at least) half framerate when content is falling behind? (At least for FxR/VR)

Comment 28

[Jean-Yves Avenard [:jya]]

(In reply to Jeff Gilbert [:jgilbert] from comment #27)
> It really sounds like we need to expose missed frames to the OS compositor.
> That's the mechanism the system is designed for, so anything else is going
> to be trickier and more fragile.

we do already (or we should)

If the compositor skips frames due to them being presented late, that amount of frames dropped is reported to the JS via the video quality API.

Comment 29

[Kelsey Gilbert [:jgilbert]]

Sure, but we need to report downward into the OS, not just to JS. Without reporting surfacing missing of frames to the OS, we'll still be downclocked.

Comment 30

[Chris Peterson [:cpeterson]]

Moving to the "Core::Audio/Video: Playback" component because this is a core Gecko issue, not a GeckoView issue.

Comment 31

[Maire Reavy [:mreavy]]

Per an email update from Lars, my understanding is this is no longer a "must have" for https://bugzilla.mozilla.org/show_bug.cgi?id=1528330. So I'm removing this bug as a dependency.

Comment 32

[Adam Stevenson [:adamopenweb]]

Does anyone know if this issue exists on the Oculus Quest?

Comment 33

[BugBot [:suhaib / :marco/ :calixte]]

Redirect a needinfo that is pending on an inactive user to the triage owner.
:jimm, since the bug has high priority, could you have a look please?

For more information, please visit auto_nag documentation.