1504886 - Twitch.tv video playback is laggy and freezes using GV 67 on Fire TV

Status: VERIFIED FIXED

(Core :: JavaScript: GC, defect, P2)

Description

[Chris Peterson [:cpeterson]]

QA reports that Twitch.tv video playback is very laggy using FFTV/GV62 but smooth with FFTV/WebView. The problem was reproducible on both the Fire TV Cube and Stick (AFTT).

https://docs.google.com/spreadsheets/d/1ZNn6a_po2QjpoMHg7fHvVGN0SY23pJBYYIfPOy91Vk0/edit#gid=1362640917&range=39:39

Comment 1

[Chris Peterson [:cpeterson]]

QA tested FFTV+GV62 on the Fire TV Cube and Stick.

Comment 2

[Vicky Chin [:vchin]]

This is a profile on my Moto G5.
https://perfht.ml/2zDbXDw

Site: https://m.twitch.tv/ninja

I notice that it's especially laggy within the first 20 seconds after page load. The lag in the video coincides with when the GC slice happens in the profile.

After about 30 seconds, the video plays fairly smooth with little or no lag. This profile was taken 40 seconds into starting the  video stream and there was just a minor lag (seems to coincide with another smaller GC slice though it's doing something different)
https://perfht.ml/2zzOuU1

Comment 3

[Vicky Chin [:vchin]]

Profile with Media thread
https://perfht.ml/2zCv8h2

It wasn't as laggy this time around though.

Comment 4

[Jim Porter (:squib)]

I profiled a Twitch clip (30 second highlight from a recording, with a replay of the chat):
https://clips.twitch.tv/PoliteSpinelessStarlingPipeHype

Here's the first profile, where I forgot to include the media thread. However, there's some *really* bad GC here:
https://perfht.ml/2DcDwqL

Here's the second, which has the media thread, but is a lot less bad:
https://perfht.ml/2DbPDoa

Comment 5

[Olli Pettay [:smaug][[e-mail]]]

That long GC slice is there because main thread has been too busy for a long time and there is pending CC, so the GC is forced to finish.
And looks like main thread is super busy because there is some animation running all the time and animationend event listener does some heavy JS.
Based on the minor GCs, the page is allocating new objects all the time.

Comment 6

[Vicky Chin [:vchin]]

Profile with lots of lag with media thread:
https://perfht.ml/2zRGjm3

Comment 7

[Chris Pearce [:cpearce (Not reading bugmail)]]

(In reply to Vicky Chin [:vchin] from comment #6)
> Profile with lots of lag with media thread:
> https://perfht.ml/2zRGjm3

All of the media threads are spending between 60-90% of their time in ThreadPool::Run::Wait, so I don't think the problem here is necessarily due to the media threads uses all the CPU time.

The Media code uses TaskQueues to dispatch tasks to a ThreadPool. Runnables to run the tasks in the TaskQueue's queue are dispatched to the target thread pool and said Runnable loops until the TaskQueue's queue of tasks is empty. The task queue can grow if other code adds another task. So I don't think unresponsive chunks of time on the media threads are indicative of a problem; they're an artifact of how media code uses thread pools.

My bet here is that Twitch's JS is doing something slow. There are long unresponsive time periods spent in timeupdate and animationend event handlers

Comment 8

[Chris Peterson [:cpeterson]]

Vicky, in today's Performance team meeting we discussed reaching out to Twitch on our mozilla-twitch-discuss list. I missed the end of the meeting. Did someone volunteer email the bug details to the list?

Comment 9

[Jim Porter (:squib)]

For what it's worth, the mobile and desktop versions of Twitch clips appear to be the same, and the only differences in visuals are because of some responsive CSS that changes the layout. For what it's worth, here are a pair of profiles so we can look at any differences more closely:

  Desktop (big window, desktop user-agent):  https://perfht.ml/2DmmMgU
  Mobile (small window, Android user-agent): https://perfht.ml/2DnA0Kf

After this, I'll take a further look at live streams, which do appear to have more differences (with a mobile user-agent, you get redirected to m.twitch.tv, which has a different layout).

Comment 10

[Vicky Chin [:vchin]]

(In reply to Chris Peterson [:cpeterson] from comment #8)
> Vicky, in today's Performance team meeting we discussed reaching out to
> Twitch on our mozilla-twitch-discuss list. I missed the end of the meeting.
> Did someone volunteer email the bug details to the list?

The issue discussed doesn't address the laggy playback here. It was simply the observation that the way Twitch plays back the chat can be optimized. Markus will send an email to the mailing list but it doesn’t affect video smoothness.

Moving this to JavaScript team for further investigation.

Comment 11

[Steve Fink [:sfink] [:s:]]

Hm, I'm not making too much of this. The profile in https://perfht.ml/2zDbXDw from comment 2 seems nicely representative, though I'm not seeing much of a correlation between event processing delay and GC slices. There's one massive 204ms slice, but all the rest are 10-12ms or so. If the GCs are lining up with visual delay, I'm not sure how to see that in the profile.

That 204ms slice is being worked on; it's some nonincremental gray marking stuff that jonco is fixing in bug 1463462.

There are certainly a whole bunch of event processing delays in the profile, mostly associated with 'timeupdate' DOMEvents. Those seem to run a bunch of JS code. I keep bumping into jit stuff when I look at the JS execution, but not in large percentages. Some of the function names sounded React-like, but I couldn't tell more than that.

jandem, what's the current state of the art in diagnosing extensive JS execution? Please forward this bug as appropriate.

Comment 12

[Chris Pearce [:cpearce (Not reading bugmail)]]

Can someone who can repro update their GV to the latest Nightly Gecko and then retest? ReadableStreams was enabled about a week ago, and when that's available Twitch will use that and it should perform better.

Comment 13

[Jan de Mooij [:jandem]]

Clearing NI; when this came up in Orlando the conclusion was there were bigger fish elsewhere.

Comment 14

[David Bolter [:davidb] (NeedInfo me for attention)]

(In reply to Steve Fink [:sfink] [:s:] from comment #11)

That 204ms slice is being worked on; it's some nonincremental gray marking
stuff that jonco is fixing in bug 1463462.

This landed about a month ago and showed a 6% win on raptor-tp6-youtube-firefox (windows)... Did it reduce the severity of this bug?

Comment 15

[Chris Peterson [:cpeterson]]

(In reply to David Bolter [:davidb] (NeedInfo me for attention) from comment #14)

This landed about a month ago and showed a 6% win on raptor-tp6-youtube-firefox (windows)... Did it reduce the severity of this bug?

Vicky, can someone your performance team test the current behavior of Twitch video in GV?

Comment 16

[Vicky Chin [:vchin]]

I tried with latest GV on the Moto G5 and Pixel 3. I can see some lag during the first 30 seconds only on the Moto G5 but not on the Pixel 3. Once I get past the first 30 seconds it's been pretty smooth.

I couldn't get a profile because the app crashes due to bug 1527228.

NI npark to test with the new FFTV+GV build.

Comment 17

[No-Jun Park [:njpark]]

Also trying to get the FFTV+GV number from local run of nimbledroid script. will find out today. Keeping the ni? flag on until then.

Comment 18

[No-Jun Park [:njpark]]

Can't test on FFTV+GV yet, since the cursor is unresponsive. https://github.com/mozilla-mobile/firefox-tv/issues/1842

Comment 19

[Chris Peterson [:cpeterson]]

(In reply to No-Jun Park [:njpark] from comment #18)

Can't test on FFTV+GV yet, since the cursor is unresponsive. https://github.com/mozilla-mobile/firefox-tv/issues/1842

Thanks. I'll make a note to retest this after FFTV+GV's cursor navigation is fixed.

Comment 20

[David Bolter [:davidb] (NeedInfo me for attention)]

Is this testable now?

Comment 21

[No-Jun Park [:njpark]]

I tried it on Fire Cube with GV 67, and it is still quite laggy. This time I see the frozen screen and spinning wheels in the center. Upon opening first twitch video, it doesn't play at all, exiting and revisiting different video makes things better, but eventually all opened videos froze and shows spinning wheels.

Comment 22

[Chris Peterson [:cpeterson]]

Nils, Twitch video playback with GeckoView on the Fire TV Cube seems have gotten worse. GV 62's video playback was laggy and GV 67 either doesn't play videos or they freeze. When will a media engineer be available to investigate?

This bug was moved to the JavaScript Engine Bugzilla component because some profiles showed long GCs, but I don't know if GC or JS is still the problem we're seeing.

Comment 23

[Vicky Chin [:vchin]]

jhlin also noticed GCs in the Cube video playback.

Comment 24

[Adam Stevenson [:adamopenweb]]

Speaking with Nils, there are some patches about to land which may help this issue. We'd like to check back in next week to confirm if the performance has improved.

Comment 25

[Nils Ohlmeier [:drno]]

(In reply to Adam Stevenson [:adamopenweb] from comment #24)

Speaking with Nils, there are some patches about to land which may help this issue. We'd like to check back in next week to confirm if the performance has improved.

Yes I think we should land bug 1531833 and then re-test.

Comment 26

[Chris Peterson [:cpeterson]]

[gvtv:p2] because this doesn't need to block GVTV MVP if it's a Twitch-specific issue. cpearce says this bug is also reproducible in Firefox desktop. Perhaps the Ion tier-4 JIT (bug 1382650l uplifted to GV 67) or the Baseline Interpreter (bug 1499324, WIP) will help?

Comment 27

[Randell Jesup [:jesup] (needinfo me)]

This should be retested on a 4K stick

Comment 28

[No-Jun Park [:njpark]]

Tested on 4K stick with Fire TV build containing 68.0.20190414095735. the playback runs great on release build, with 900p60fps. It runs a bit slower on debug build, but I think that is expected. Sorry that I didn't get to it earlier.

Comment 29

[Nils Ohlmeier [:drno]]

(In reply to No-Jun Park [:njpark] from comment #28)

Tested on 4K stick with Fire TV build containing 68.0.20190414095735. the playback runs great on release build, with 900p60fps. It runs a bit slower on debug build, but I think that is expected. Sorry that I didn't get to it earlier.

Thank you for checking and for bringing back such good news!

It looks like the patches we landed in other bugs to improve playback performance on Fire TV have resolved the issue then. I'll close this. If anyone disagrees and thinks there is still work left to be done feel free to re-open the bug.