Low frame rate, high CPU usage when playing full-screen Flash video with Twitch's new HTML video controls

RESOLVED FIXED

Status

()

Core
Graphics: Layers
RESOLVED FIXED
3 years ago
2 years ago

People

(Reporter: cpeterson, Unassigned)

Tracking

(Blocks: 1 bug, {flashplayer})

unspecified
flashplayer
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(platform-rel -)

Details

(Whiteboard: [platform-rel-Amazon][platform-rel-Twitch], URL)

(Reporter)

Description

3 years ago
Twitch users on Reddit are reporting low frame rate with full-screen Flash video in Firefox but not Chrome. The problem seems to have started when Twitch launched their new HTML video controls, which is an HTML overlay that controls the Flash Player playing the Flash video.

https://www.reddit.com/r/Twitch/comments/3n3ull/terrible_fullscreen_fps_with_the_new_html5_player/

1080p streams dropped from 50 fps to 30 or less, with the plugin-container using ~70% CPU. Someone said turning off Protected Mode in Flash's settings fixed their issues.

Comment 1

3 years ago
URL:  http://www.twitch.tv/monstercat

Firefox  41 & Flash Player 19:
1280 x 720P video
~30 FPS
11 skipped frames (per Twitch stats)

Chrome 45 & Flash Player 19:
1280 x 720P video
~30 FPS
3034 skipped frames (per Twitch stats)

It's unclear if Twitch will actually serve more than 30 FPS for this stream.

Comment 2

3 years ago
Also...

Firefox  41 & Flash Player 19:
Memory usage: 48MB (per Twitch stats)

Chrome 45 & Flash Player 19:
Memory usage: 187MB (per Twitch stats)

Comment 3

3 years ago
URL:  http://www.twitch.tv/riotgames

Firefox  41 & Flash Player 19:
1280 x 720P video
~54 FPS
382++ skipped frames about 1 per second (per Twitch stats)
Memory usage: 78MB (per Twitch stats)

Chrome 45 & Flash Player 19:
1280 x 720P video
~50 FPS
480++ skipped frames about 1 per second (per Twitch stats)
Memory usage: 212MB (per Twitch stats)

Comment 4

3 years ago
URL:  http://www.twitch.tv/summonersinnlive/v/18635199 (fullscreen)

Firefox  41 & Flash Player 19:
1280 x 720P (source) video
~60 FPS (embedded)
~45 FPS (fullscreen)

Chrome 45 & Flash Player 19:
1280 x 720P (source) video
~60 FPS (embedded)
~60 FPS (fullscreen)

I'm seeing quite a bit of invalidation in the invisible playback control area (using paint flashing) in and out of fullscreen.
(Reporter)

Comment 5

3 years ago
Daniel, does this sound like a layout problem? Can you recommend someone to look at this?

Twitch just launched their new video player (Flash video + HTML video control overlay) and Firefox video frame rate has regressed. Like Jet says in comment 4, the HTML control area is invalidated once or twice a second even when Flash is playing full-screen video.

Some notes from a Twitch web developer:

Flash Stage3D is no longer an option with this new player because Stage3D requires the swf to be embedded with wmode "direct". This causes Flash to draw directly to the screen buffer, hiding our HTML UI layered on top. We have to use wmode "transparent" which has worse performance and disables hardware acceleration.

Wierdly enough, direct doesn't hide the UI in just PPAPI Flash (Chrome). It also gets better performance despite rendering the exact same way.

We noticeably improved performance by switching a simple overlay from opacity 0 to display none (5 fps, far more than expected). I'm going to experiment again but I think there's something fundamentally slow with laying HTML on top of Flash. Going from a maximized window to full-screen drops maybe 20 fps which makes no sense either.
Flags: needinfo?(dholbert)
Poked around a little in devtools inspector. Found out so far: this element (or rather its descendants)...
  <div class="player-hover player-controls-bottom js-controls-bottom">
... seems to be responsible for some/all of the invalidation over the video.

More when I'm back from lunch.
[methodology: I've got paint flashing on, & I've been marking groups of elements with "display:none", and then disabling that on some of the them, to see which trigger the periodic painting]

Smallish issues:  there are three elements that cause us to repaint a *small* area, even when the page has them hidden (from the video being not-hovered):
===============
 1) <div class="player-seek__time js-seek-currenttime ...">
  * This is the "current time" ticking digital clock that's superimposed on the left side of the video.

 2) <span class="player-slider__buffer js-seek-buffer">
 3) <div class="ui-slider-range ...">
  * These each draw parts of  draw the time "scrubber bar", showing the current time and how much beyond the current time has been buffered. These elements' updates trigger miniscule repaints at the sliver of the scrubber-bar where time is currently passing.

Largish issue: This is the element that makes us repaint the whole video area periodically:
==============
  <a class="ui-slider-handle ui-state-default ui-corner-all" href="#"></a>

This element (actually its ::before-generated child) is the circular "handle" that appears over the time scrubber-bar when I hover over it. It's absolutely positioned, and its "left" property is dynamically tweaked with JS.  Though I still see large repaints even if I add give this element "left: 20px !important" to keep the page's dynamic left-tweaks from having any effect on computed 'left'.  So I'm not sure what causes the repaints.  ("display:none" on this <a> makes the large repaints stop, though.)
Depends on: 1211654

Comment 8

3 years ago
oops-i-meant-bug-1211654
I think it's reasonable to call this largish issue a Layers bug, given the repaint behavior I'm seeing in comment 7.

I filed bug 1210899 with what I think is a reduced testcase for the "Largish issue" with the slider handle forcing extra repainting. (The smallish issue may be the same underlying gecko bug, too; not sure.)
Component: Plug-ins → Graphics: Layers
Flags: needinfo?(dholbert)
(Reporter)

Comment 9

3 years ago
Thanks, Daniel! I will forward this information to Twitch. Maybe they can work around the problems you've identified until "opacity:0" bug 1211654 is resolved.
Cool! FWIW, the best way for them to work around this in the meantime is probably to use "display:none" on the overlay when it's not supposed to be visible.

(That's admittedly a bit tricky to get working without breaking their transition:opacity "fade in" / "fade out" effect that they've got on the controls-overlay.  For this to work, they'd need to basically do the following:
 - When they want the controls to become visible [i.e. when they set the data-controls="true" attribute on their <div> with class="player"], remove "display:none" *immediately*.
 - When they want the controls to become hidden [i.e. when they set the data-controls="false"], register a transitionend handler to set "display:none".  This handler will fire when the opacity fade-out transition has completed, so they won't be stomping on the fade-out animation.
)
OK, I think I have a better workaround.

As mattwoodrow noted in bug 1211654, what's basically happening here is that we're oscillating between making the controls area an active layer (because stuff's changing), and then demoting it (because nothing's changed for 250ms, so it looks like it's no longer animating).  And there's a nontrivial painting cost for promoting & demoting an active layer.

So, clearly we need to tune our layerization heuristics (probably in bug 1211654) for stuff like this to "just work" efficiently. But as a better workaround, Twitch.tv can force the controls area to *remain* an active layer by styling an ancestor of all of the changing stuff with "will-change: opacity".

So e.g.:
  .player-controls-bottom { will-change: opacity }
...or more targeted:
  .player-slider { will-change: opacity }

I verified that these rules solve the frequent invalidation in the controls-area that Jet mentioned in comment 4, in Firefox Nightly (with e10s).  Not sure if it will help with Firefox release version though, but I'm hope it will.
FWIW: unfortunately bug 1211654 didn't fix the excess invalidation here, as far as I can tell. (There's still a bunch of extra invalidation every every time the [invisible] scrubber "handle" moves). I tried poking around with devtools to see if I could create a minimal testcase that triggers the same issue, but haven't had luck yet (in part because the scrubber moves due to javascript, and if I modify the page too much, the javascript trips over something & stops working).

So, for now, the best solution is the more targeted "will-change" hint that I suggested in comment 11:
  .player-slider { will-change: opacity }

That does seem to fix the issue, at least as far as paint flashing can tell.
For demonstration purposes, here's a user-style which is installable with the "Stylish" add-on & which fixes the problem for me, by locally applying the CSS change suggested in comment 11:
  https://userstyles.org/styles/120153/twitch-tv-will-change-hint-for-bug-1210899
(Reporter)

Comment 14

3 years ago
Twitch says they are testing Daniel's CSS workaround. If it improves the Flash frame rate problems, they will push it to production later this week.
I see this in their stylesheet now, FWIW:
http://player.twitch.tv/css/player.css
.player-hover{opacity:0;-webkit-transition:opacity .2s ease-out;transition:opacity .2s ease-out;will-change:opacity}

That style applies to the element mentioned in comment 6 (that's the subtree for the control that appear on hover) which means that probably is a decent place to layerize.

And great news -- I can't reproduce any bad invalidation now (in Nightly at least, haven't tested release) -- and if I disable that "will-change" decl in devtools, then I see the excess invalidations come back (visible w/ paint flashing). So that does seem to be what fixed it.
Chris, could you circle back on the reddit thread -- or with twitch staff -- to find out if the frame-rate issue is fixed?

(I was never able to directly reproduce that, so I've been using unnecessary-paints-revealed-by-paint-flashing as my undesirable "actual results" here, and that's what the "will-change" workaround has addressed. I'm hoping that fixes the low framerate, too, but I can't be sure.)
Flags: needinfo?(cpeterson)
(Also: I spun off bug 1231946 to cover the unnecessary repaints, which the site no longer triggers by default but which is still reproducible if you disable the "will-change" hackaround.)
Blocks: 1231946
(Reporter)

Comment 18

3 years ago
(In reply to Daniel Holbert [:dholbert] from comment #16)
> Chris, could you circle back on the reddit thread -- or with twitch staff --
> to find out if the frame-rate issue is fixed?

I emailed our Twitch contact.
Flags: needinfo?(cpeterson)
(Reporter)

Comment 19

3 years ago
Twitch says, "We deployed the suggested CSS fix. It definitely reduced page redraws, although I don't know how much it impacted Flash video FPS."

On my MacBook Pro, the Twitch Flash player can play a (supposedly) 60-fps stream at about 35 fps fullscreen in both Firefox and Chrome. I'm going to resolve this bug as fixed by Twitch. Bug 1231946 is still open to track the unnecessary paint issue that was worked around by the CSS opacity change.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
Actually, I suspect bug 1237083 was the real main issue here that was causing user complaints (but I didn't notice it was a bug & file/isolate it until this week).  It only happens with e10s disabled, and Firefox users generally do have e10s disabled.

So -- for remaining low-framerate issues on Twitch, see that bug.
Blocks: 1237083
Comment hidden (me-too)

Comment 22

3 years ago
If you have a 4K monitor you are super much affected by this.
Thanks, Art & Ernst; it's useful to know users are still affected by this. I suspect your issue is covered by bug 1237083, as noted in comment 20.  Let's move further discussion on remaining twitch.tv repaint issues to that bug.

(Also, since Art's comment 21 was quite tall, and the about:support information in that comment isn't actually needed, I've tagged it as "me-too", just to make bugzilla auto-collapse it. No offense intended; just aiming to keep this bug page as readable/skimmable as possible.)
(Reporter)

Comment 24

3 years ago
Art, I think the problem is that Firefox on Linux does not hardware-accelerated compositing ("accelerated layers"). Firefox on OS X and Windows supports hardware-accelerated compositing. If I disable it by setting the "layers.acceleration.disabled" pref = false, then Twitch's frame rate on my Mac drops from ~30 fps (same as Chrome for me) to ~10 fps. So I don't think there are any more optimizations we can do for Twitch video until Linux supports accelerated layers.

I think bug 594876 is tracking the work to support accelerated layers on Linux. You might try testing "layers.acceleration.disabled" pref = true on your machine, BUT that might cause stability problems, depending on your hardware!

Comment 25

3 years ago
Chris, I have this problem both on Windows 10 and Linux with OMTC + layers.acceleration.force=true
(Reporter)

Updated

3 years ago
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Ernst:
 (1) The pref name is actually "layers.acceleration.force-enabled" (comment 25 was missing "-enabled") -- can you double-check that this is the pref you're using?

 (2) (Don't try this if you happen to have flash-triggered epilepsy):
     Could you enable paint flashing (set nglayout.debug.paint_flashing = true) and then reproduce the slowness, with accelerated layers force-enabled, and let us know what the video looks like? In particular: does the video area stay the same hue, or does it randomly flash between different hues as in attachment 8704788 [details]?
Flags: needinfo?(ernstp)

Comment 27

3 years ago
1) Yes I mean force-enabled!

Version: 46.0a1
Build ID: 20160111030207
Update Channel: nightly
Multiprocess Windows: 1/1 (default: true)
GPU Accelerated Windows: 1/1 OpenGL (OMTC)
Supports Hardware H264 Decoding: No;
Vendor ID: X.Org
WebGL Renderer: X.Org -- Gallium 0.4 on AMD FIJI (DRM 3.1.0, LLVM 3.8.0)
windowLayerManagerRemote: true
AzureCanvasBackend: cairo
AzureContentBackend: cairo
AzureFallbackCanvasBackend: none
AzureSkiaAccelerated: 0
CairoUseXRender: 0
Flags: needinfo?(ernstp)
OK, thanks for confirming that. Did you look into part (2) of comment 26? That's the more-interesting question there...
Flags: needinfo?(ernstp)

Comment 29

3 years ago
Hi!
For 2) the video area doesn't get any repaint color at all, it stays normal. The page around it and the Twitch overlays etc flash though. Same on Linux & Windows 10.
I have similarly bad performance in Windows 10 btw.
Flags: needinfo?(ernstp)
Hm, ok - thanks. That may mean you're seeing something distinct from bug 1237083, then.

Comment 31

3 years ago
Hey.
I'm suffering from effects of this bug by having low fps with custom css which forces chat to be drawn on top of video (semi-transparent).
Playing 60fps VoD without css 1-5 frames dropped each second. With css up to 30.
On Chrome 47.0.2526.111 with this css (just lacks a few position fixes) 1 frame dropped per 3-5 seconds.
CSS: http://pastebin.com/wkNcVndQ
BetterTTV (DarkenTTV option) + Rechat (for VoDs) recommended in case of test of css.

Comment 32

2 years ago
Any update on this?

Comment 33

2 years ago
Still experiencing this bug, especially when I'm trying to display semitransparent chat on top of video.

Comment 34

2 years ago
I'm also experiencing similar behavior. The playback drops from 30fps to 20fps on cinematic mode and to 15fps on full screen.

Comment 35

2 years ago
I am also experiencing this exact behavior. Video is just fine (actually, fantastic) when using the normal Twitch window with menus on the left hand side and chat on the right hand side. However, when going full screen, the FPS drops to roughly 15 FPS from anywhere around 30-60 FPS in the normal mode. In addition, a slight audio delay relative to the video seems to be introduced when going to full screen.

Comment 36

2 years ago
I confirm the problem too. The video is ok at original size, but the FPS drop very low when I turn on fullscreen on Twitch.
platform-rel: --- → ?
Whiteboard: [platform-rel-Amazon][platform-rel-Twitch]

Comment 37

2 years ago
Note that Twitch has now a new HTML5 player.
It seem to run ok at fullscreen on Linux for me.
Chris, can you re-test and close if it works well for you, too?
Flags: needinfo?(cpeterson)

Comment 39

2 years ago
Hello guys. I have just tested the player. For the moment i work well with HTML5 and the old flash player! No difference between them. :)
(Reporter)

Comment 40

2 years ago
Thanks for testing, Burno and Bogdan! I was not able to reproduce the original problem, so your testing is a big help. :)

I'm marking the platform-rel flag to - because Twitch is moving from Flash to HTML5 video, so there is no need to reach out to them at this time.
Status: REOPENED → RESOLVED
Last Resolved: 3 years ago2 years ago
platform-rel: ? → -
Flags: needinfo?(cpeterson)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.