1265553 - Yahoo Music is slow/choppy when e10s is enabled

Status: RESOLVED WORKSFORME

(Core :: Layout, defect, P3)

Description

[u554753]

Yahoo Music is slow/choppy when e10s and hardware acceleration is enabled

[Pre-Requisites]
e10s is enabled
Hardware Acceleration can be either enabled or disabled

Steps to Reproduce:
1. Launch Firefox.	
2. Navigate to https://www.yahoo.com/music/	
3. Navigate to a link that plays music 	

Expected Results:
Audio and video play smoothly.

Actual Results:
Audio and video lags and is choppy.

The bug was encountered on the following hardware:
(1) 
Windows 10 x32 w/FFx32
Acer Switch 10E					
PROCESSOR: Z3735F@1.33GHz		
RAM: 2GB		
GRAPHICS: Intel HD Graphics Direct 3D11		
BUILD ARCH: Version 10.0 - 10240		

(2) 
Windows 10 x64 w/FFx32
HP Notebook			
PROCESSOR: AMD A6-5200 APU @2.00GHZ		
RAM: 4GB / 3.45 USABLE		
GRAPHICS: AMD Radeon HD 8400 / R3 Series Direct3D11		
BUILD ARCH: Version 10.0 - 10240	

(3)
Windows 10 x64 w/FFx32
Acer Aspire				
PROCESSOR: N3050@1.60GHz		
RAM: 2GB		
GRAPHICS: Intel HD Graphics Direct3d11 		
BUILD ARCH: Version 10.0 - 10240		

(4)
Windows 8 x64 w/FFx32				
PROCESSOR: AMD A4-6210 APU @1.80ghZ		
RAM: 4GB		
GRAPHICS: AMD Radeon R3 Graphics Direct 3D11		
BUILD ARCH: Version 6.3 - 9600	

(5)
Windows 10  x64 w/FFx32			
PROCESSOR: Intel Core i3-4005U @1.70GHz		
RAM: 8GB		
GRAPHICS: Intel HD Graphics Family Direct 3D11		
BUILD ARCH: Version 10.0 (Build 10240)

Comment 1

[Chris Pearce [:cpearce (Not reading bugmail)]]

JW: Would you be able to look into this?

Comment 2

[JW Wang [:jwwang] [:jw_wang]]

Checking...

Comment 3

[JW Wang [:jwwang] [:jw_wang]]

Can you catch logs with |NSPR_LOG_MODULES=timestamp:1,MediaDecoder:4,AudioStream:3|? Thanks!

Comment 4

[u554753]

Attachment: Log for Bug 1265553

Please let me know if these are the logs you need. Thanks.

Comment 5

[JW Wang [:jwwang] [:jw_wang]]

The logs said the duration is about 15s.

Some queustions:
1. Did the glitch happen all the time during playback or only at the end of playback?
2. It is suspicious that the decoder went into dormant state before reaching the end. Can you record a video to show how it is like when glitch happens?

Comment 6

[JW Wang [:jwwang] [:jw_wang]]

Btw, I can't repro it on my Windows 10 laptop. Can you give the specific link with choppy music? Is it 100% reproducible on a specific link?

Comment 7

[u554753]

Hi, here is a video recording requested per Comment 5. It seems that this behavior is unpredictable, but do keep in mind the specs provided above were for testing with low-end machines specifically. In this case, the advertisements played at the beginning were skipping and choppy whereas the movie played afterward was a bit better in quality. 

https://gwimberly-softvision.tinytake.com/sf/NjI0MTg3XzMwNzE4ODI

Comment 8

[JW Wang [:jwwang] [:jw_wang]]

(In reply to Grover Wimberly IV from comment #7)
> Hi, here is a video recording requested per Comment 5. It seems that this
> behavior is unpredictable, but do keep in mind the specs provided above were
> for testing with low-end machines specifically. In this case, the
> advertisements played at the beginning were skipping and choppy whereas the
> movie played afterward was a bit better in quality. 
> 
> https://gwimberly-softvision.tinytake.com/sf/NjI0MTg3XzMwNzE4ODI

The cpu (@1.3GHz) should be still powerful enough to play the video. I wonder if it suffers the same issue as bug 1195767 comment 1195767.

Hi Jya,
Can you provide instructions to turn off hardware-acceleration decoding on Windows?

Hi Grover,
Can you provide the link which you played in the recording?

Comment 9

[Jean-Yves Avenard [:jya]]

media.hardware-video-decoding.enabled=false

that is on recent version, prior it was another pref. Must restart after changing this pref.

Comment 10

[JW Wang [:jwwang] [:jw_wang]]

Hi Grover,
Can you turn off hard acceleration to see if that will make a difference?

Comment 11

[u554753]

Hi JW Wang,

I turned off hardware acceleration in the manner provided in Comment 6. Ads played at the beginning of the video were still choppy, but the actual music didn't seem to skip any.

The link I listened to was this: https://www.yahoo.com/music/song-premiere-hear-voice-champ-sawyer-151113244.html

Comment 12

[u554753]

Sorry, meant Comment 9, not 6.

Comment 13

[JW Wang [:jwwang] [:jw_wang]]

According to comment 11, it seems there isn't much we can do here.

The CPU just isn't powerful enough to give you smooth video playback. The best shot you can give is turn off hardware acceleration so at least you have a smooth audio playback.

Hi Jya,
Do you have other tips to improve playback?

However, I am still surprised by bug 1195767 comment 51 where GPU becomes a bottleneck of CPU.

Comment 14

[Mike Conley (:mconley) (:⚙️)]

Out of curiosity, how does Firefox perform with e10s disabled on these sites?

Comment 15

[u554753]

Hi Mike,

It works fine with e10s disabled. A detailed graph of e10s/Hardware compatibility for Yahoo music with these configurations I posted can be accessed here.

https://docs.google.com/a/mozilla.com/spreadsheets/d/1eH9O1PTrg6oKpHzd4B4520D6AfbQIxK54E5YVkQ8pug/edit?usp=sharing

Comment 16

[Mike Conley (:mconley) (:⚙️)]

If things work fine with e10s disabled, then it seems that the CPU is powerful enough to do the job - I think the bottleneck must be elsewhere.

Does the fact that non-e10s works without issue change things for you, jwwang?

Comment 17

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

Can you try the latest Firefox 47 build (from http://beta.mozilla.org ) and see if this is still a problem?

Comment 18

[Jean-Yves Avenard [:jya]]

wfm

Comment 19

[JW Wang [:jwwang] [:jw_wang]]

(In reply to Mike Conley (:mconley) - (Away until June 29th) from comment #16)
> If things work fine with e10s disabled, then it seems that the CPU is
> powerful enough to do the job - I think the bottleneck must be elsewhere.
> 
> Does the fact that non-e10s works without issue change things for you,
> jwwang?

I have no idea. I set up a VM to throttle CPU frequency. But I couldn't find a frequency at which non-e10s playback is smooth while e10s one is stutter. They both showed the same smoothness/stutterness at various CPU frequency.

Hi Grover,
Can you compare the CPU/memory usage charts between e10s and non-e10s playback? Or even get profiling data so we have better idea where CPU time is spent?

Comment 20

[u554753]

Hi JW Wang,

We tested with e10s/non-e10s on a couple of low-end machines and here are some results:

Windows 10 on Acer Aspire:
PROCESSOR:  N3050@1.60GHz  
RAM:  2GB  
GRAPHICS:  Intel HD Graphics Direct3d11   
BUILD ARCH:  Version 10.0 - 10240
Notes: I observed almost same CPU usage in both e10s enabled and disabled(between 20 to 80 %)

AMD A4-6210 APU @1.80ghZ
RAM: 4GB  
AMD Radeon R3 Graphics Direct 3D11  
Version 6.3 - 9600
Notes: CPU usage almost identical going between 20 and 80%

Comment 21

[u554753]

If you need further testing, please let me know.

Comment 22

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

(In reply to Grover Wimberly IV [:Grover-QA] from comment #21)
> If you need further testing, please let me know.

Can you paste in the graphics section of about:support for one of the machines so we can figure out whether we have accelerated layers and hardware decode.

Comment 23

[Liz Henry (:lizzard) (relman/hg->git project)]

Tracking for 48+ since this looks like a recent regression affecting e10s.

Comment 24

[u554753]

(In reply to Anthony Jones (:kentuckyfriedtakahe, :k17e) from comment #22)
> (In reply to Grover Wimberly IV [:Grover-QA] from comment #21)
> > If you need further testing, please let me know.
> 
> Can you paste in the graphics section of about:support for one of the
> machines so we can figure out whether we have accelerated layers and
> hardware decode.

Hi Anthony, 

Here is the info from the Graphics of about:support for the Acer Aspire machine.

Compositing    Direct3D 11
Asynchronous Pan/Zoom    wheel input enabled; touch input enabled
WebGL Renderer    Google Inc. -- ANGLE (Intel(R) HD Graphics Direct3D11 vs_5_0 ps_5_0)
Hardware H264 Decoding    Yes; D3D11 blacklisted with DLL igd10iumd32.dll (10.18.15.4248); Using D3D9 API
Direct2D    true
DirectWrite    true (10.0.10586.0)
GPU #1
Active    Yes
Description    Intel(R) HD Graphics
Vendor ID    0x8086
Device ID    0x22b1
Driver Version    10.18.15.4248
Driver Date    8-4-2015
Drivers    igdumdim64 igd10iumd64 igd10iumd64 igd12umd64 igdumdim32 igd10iumd32 igd10iumd32 igd12umd32
Subsys ID    106d1025
RAM    Unknown

Comment 25

[Jim Mathies [:jimm]]

relman is tracking this for 48 and beyond so not adding this to an e10s mileston.

Anthony, is there someone on your team that can take a look? This is outside the areas of expertise of the current core e10s team.

Comment 26

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

Grover,

This page seems to have a bunch of image animations going below if you scroll down then back up again. I'm not sure if that is part your STR but I saw some scrolling at the beginning of the video. We can figure out if image animation is involved by setting image.animation_mode=none in about:config and then reloading the page. You should be able to observe all the animations no longer happening.

Comment 27

[u554753]

Hi Anthony,

With image.animation_mode=none, the movie is still choppy when scrolling up and down the page and particularly when the video is loading or showing miscellaneous advertisements. When the normal video plays, it is not choppy.

Comment 28

[JW Wang [:jwwang] [:jw_wang]]

Per comment 19, I have no idea how to progress this bug. Unassign myself so someone can pick it up.

Comment 29

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

Jim - I don't think this is a video related issue. My assessment is that it is fundamentally caused by e10s painting a larger area.

Comment 30

[Jim Mathies [:jimm]]

Over to milan then since this might be gfx related.

Comment 31

[Milan Sreckovic [:milan] (needinfo for best results)]

I'm tempted to agree with Anthony and just say "these computers can't keep up".  Some things are interesting:

* Turning off hardware accelerated video decoding improves things (although on Acer Aspire, it shouldn't make a difference as we already block that functionality.)
* The image animation seems to be the biggest issue
* Turning E10S off helps things
* One about:support we have points to otherwise accelerated setup

Sotaro, can you think of something obvious that fits the bill before we go with the "too slow of a computer" theory?

Comment 32

[Sotaro Ikeda [:sotaro]]

advertisements seems like a bit different problem. When I tested Yahoo Music, all advertisements are shown by using flash.

Comment 33

[Sotaro Ikeda [:sotaro]]

Somehow, part of page is continuously repainted forever. It happens even when video playback is paused. It could be checked with "nglayout.debug.paint_flashing:true".

Comment 34

[Sotaro Ikeda [:sotaro]]

(In reply to Sotaro Ikeda [:sotaro] from comment #33)
> Somehow, part of page is continuously repainted forever. It happens even
> when video playback is paused. It could be checked with
> "nglayout.debug.paint_flashing:true".

This painting seems to increase cpu usage.

Comment 35

[Sotaro Ikeda [:sotaro]]

Attachment: Call stack of the invalidation

This is a call stack when the invalidation happens. Somehow VectorImage continues to invalidate a frame forever.

Comment 36

[Sotaro Ikeda [:sotaro]]

(In reply to Sotaro Ikeda [:sotaro] from comment #35)
> This is a call stack when the invalidation happens. Somehow VectorImage
> continues to invalidate a frame forever.

Actually, the VectorImage loaded the following.
 https://s.yimg.com/os/yc/pt/icons/fuji-loader-v2-blue.0.svg

Comment 37

[Sotaro Ikeda [:sotaro]]

(In reply to Sotaro Ikeda [:sotaro] from comment #36)
> (In reply to Sotaro Ikeda [:sotaro] from comment #35)
> > This is a call stack when the invalidation happens. Somehow VectorImage
> > continues to invalidate a frame forever.

It seems like layout or svg related problem.

Comment 38

[Jet Villegas (inactive)]

(In reply to Sotaro Ikeda [:sotaro] from comment #37)
> (In reply to Sotaro Ikeda [:sotaro] from comment #36)
> > (In reply to Sotaro Ikeda [:sotaro] from comment #35)
> > > This is a call stack when the invalidation happens. Somehow VectorImage
> > > continues to invalidate a frame forever.
> 
> It seems like layout or svg related problem.

Tested this link:
https://www.yahoo.com/music/chubby-checker-joins-hall-oates-onstage-to-do-193119321.html

I don't see the invalidation described here with paint-flashing enabled. Can you share the actual link you're testing with?

Comment 39

[Sotaro Ikeda [:sotaro]]

I used https://www.yahoo.com/music/joel-benji-madden-wont-perform-as-good-charlotte-after-age-40-201312940.html

Comment 40

[Sotaro Ikeda [:sotaro]]

(In reply to Jet Villegas (:jet) from comment #38)
> (In reply to Sotaro Ikeda [:sotaro] from comment #37)
> > (In reply to Sotaro Ikeda [:sotaro] from comment #36)
> > > (In reply to Sotaro Ikeda [:sotaro] from comment #35)
> > > > This is a call stack when the invalidation happens. Somehow VectorImage
> > > > continues to invalidate a frame forever.
> > 
> > It seems like layout or svg related problem.
> 
> Tested this link:
> https://www.yahoo.com/music/chubby-checker-joins-hall-oates-onstage-to-do-
> 193119321.html
> 
> I don't see the invalidation described here with paint-flashing enabled. Can
> you share the actual link you're testing with?

Hmm, it is wired, I saw the flash in the above link.

Comment 41

[Sotaro Ikeda [:sotaro]]

:jet, did you enabled flash plugin? When I checked flash plugin was "Always Active". When I set the flash plugin to "Never Active", I did not see advertisement and I did not saw the the continuing paint flashing.

Comment 42

[Sotaro Ikeda [:sotaro]]

When I disabled flash plugin, I did not see the continuous paint flashing, but VectorImage::RequestRefresh() is still called frequently.

Comment 43

[Daniel Holbert [:dholbert]]

I see the continuous invalidation. For me, it seems to be an interaction between a flash plugin and this pseudo-element:

.desktop body::after,.desktop body:after {
[...]
	background:url(https://s.yimg.com/os/yc/pt/icons/fuji-loader-v2-blue.0.svg) center center no-repeat;
	background:url(https://s.yimg.com/os/yc/pt/icons/throbber-93.gif) center center no-repeat\9;
	background-size:52px 52px;
[...]
}
body::after {
    opacity: 0;
    content: ' ';
    position: fixed;
[...]
}

So, we've got a 0-opacity pseudo-element, with an animated SVG and/or GIF background. We're ticking its internal counter (and triggering invalidations) using "requestRefresh", but we should not be doing that.

I'm guessing the flash plugin is interacting here by fooling us into thinking that this 0-opacity thing actually composites together with some flash that's underneath it, for some reason.

Comment 44

[Daniel Holbert [:dholbert]]

Note the "no-repeat\9" at the end of the second "background" decl -- that makes that declaration invalid/rejected, though our devtools don't recognize that. I'll file a bug separately on that.  If I remove the "\9", then we successfully parse that line of CSS, and we use the animated GIF as the background image instead of the animated SVG, and you can see the frequency of the paint-flashing changes as a result. [It drops to the animation frequency of the GIF.]

So: don't be fooled by the apparent SVG/VectorImage thrashing here -- we have the same issue if we use the page's animated GIF, but we just thrash with a lower frequency.

The main issue here seems to be that we think that painting in the "opacity:0" fixed-position ::after element is important and worthy of receiving invalidations, and we probably incur some churn from recompositing that together with the flash plugin.  I believe this is similar to (or the same as) bug 1231946.

Comment 45

[Daniel Holbert [:dholbert]]

(I filed bug 1287620 on making devtools reflect reality around the \9 CSS hack there.)

Comment 46

[Jim Mathies [:jimm]]

should we prioritize investigating the invalidation problem here?

Comment 47

[Jet Villegas (inactive)]

(In reply to Jim Mathies [:jimm] from comment #46)
> should we prioritize investigating the invalidation problem here?

Yep, to Matt for a look.

Comment 48

[Matt Woodrow (:mattwoodrow)]

I've attached patches to bugs 1237097 and 1237102 which fix the invalidation issues for me.

Comment 49

[Gerry Chang [:gchang]]

Hi :Grover,
Can you help to verify if the issue still exists?

Comment 50

[u554753]

Hello, I re-ran the tests on two lower-end machines. If you need more, I can provide more tests as necessary.

Tested with the following specs: 
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:51.0) Gecko/20100101 Firefox/51.0

URLs Tested:
https://www.yahoo.com/music/exclusive-backstage-interview-dr-teeth-000000951.html
https://www.yahoo.com/music/weekend-3-12-16-part-095830865.html

Windows 10 x64 w/FFx32
HP Notebook			
PROCESSOR: AMD A6-5200 APU @2.00GHZ		
RAM: 4GB / 3.45 USABLE		
GRAPHICS: AMD Radeon HD 8400 / R3 Series Direct3D11		
BUILD ARCH: Version 10.0 - 10240	

On a fresh profile: Advertisements continue to play slow and are choppy. After the advertisements are done, the selected video plays and sounds smoothly.


Windows 10 x64 w/FFx32
Acer Aspire				
PROCESSOR: N3050@1.60GHz		
RAM: 2GB		
GRAPHICS: Intel HD Graphics Direct3d11 		
BUILD ARCH: Version 10.0 - 10240	

On a fresh profile: Advertisements continue to play slow and are choppy. After the advertisements are done, the selected video plays and sounds smoothly.

Comment 51

[Gerry Chang [:gchang]]

Hi Jim,
It seems that the advertisements continue to play slow and are choppy with these 2 fixes of bug 1237097 and 1237102. Can you help to find someone on this?

Comment 52

[Jim Mathies [:jimm]]

I'm not sure what to do with this, we thought this had something to do with invalidation. We suspected a couple bugs would help, those are now fixed.

Remaining issues are that some advertisements play "choppy" on low end systems while video plays well. Not sure if we really care about choppy ads. Jet, any suggestions here?

Comment 53

[Liz Henry (:lizzard) (relman/hg->git project)]

If music playback is good, I think that is enough at least for the 49 timeframe. It sounds like the major issue here is fixed but I'll leave aurora as "fix optional" in case we come up with a solution for the remaining choppiness.

Comment 54

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Hi Jim, do we still consider this one a P1? I was reviewing all P1 bugs that are tagged as status-firefox50=fix-optional and noticed this one. Given that we have fixed majority of the choppiness problems, should we lower this to a P2?

Hi Matt, if the fixes from bugs 1237097 and 1237102 (both fixed in Fx51) have helped here, should we consider uplifting them to Fx50? Or are these too risky?

Comment 55

[Matt Woodrow (:mattwoodrow)]

We can uplift them if you'd like, they were both fairly simple and low risk.

I didn't think they actually fixed the underlying bug here though, so there might not be much point.

Comment 56

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

(In reply to Matt Woodrow (:mattwoodrow) from comment #55)
> We can uplift them if you'd like, they were both fairly simple and low risk.
> 
> I didn't think they actually fixed the underlying bug here though, so there
> might not be much point.

Thanks Matt! Ok perhaps I misread some of the previous comments. Let's not uplift if it isn't helping anything.

Comment 57

[Jim Mathies [:jimm]]

(In reply to Ritu Kothari (:ritu) from comment #54)
> Hi Jim, do we still consider this one a P1? I was reviewing all P1 bugs that
> are tagged as status-firefox50=fix-optional and noticed this one. Given that
> we have fixed majority of the choppiness problems, should we lower this to a
> P2?
> 
> Hi Matt, if the fixes from bugs 1237097 and 1237102 (both fixed in Fx51)
> have helped here, should we consider uplifting them to Fx50? Or are these
> too risky?

No I would consider this a P1, and since the str hasn't really changed and we can still reproduce the choppy ad display, it should stay open. Downgrading to reflect this.

Comment 58

[Jim Mathies [:jimm]]

s/would/wouldn't

Comment 59

[Milan Sreckovic [:milan] (needinfo for best results)]

Jet, do you want to keep tracking this?  Do we have somebody that would look at it?

Comment 60

[Jet Villegas (inactive)]

Grover: Thanks for testing on the low-end machines. I'd like to see if we've actually fixed the invalidation issues found using the steps noted in comment 34. That is, try FF50 and FF51 and see if we're actually painting less with the fixes in 51.

Comment 61

[u554753]

The video is painting much less in FF50 and 51.

Comment 62

[Desigan Chinniah [:cyberdees] [:dees] [London - GMT]]

Based on Comment 61 - should this one be closed out, :jet?

Comment 63

[Jet Villegas (inactive)]

Marking this WORKSFORME based on comment 61.