1198368 - [e10s] [ADBE 4044716] Playing Amazon Instant Video using Flash in NPAPI sandbox hangs the chrome process

Status: RESOLVED WORKSFORME

(Core Graveyard :: Plug-ins, defect)

Description

[Brad Lassey [:blassey] (use needinfo?)]

Separating out this bug to investigate why a plugin hang is able to hang the chrome process under e10s, but not with e10s disabled


+++ This bug was initially created as a clone of Bug #1197943 +++

[Tracking Requested - why for this release]:

This is a regression from bug 1185532, which enabled the NPAPI sandbox for Flash by default. If I change the "dom.ipc.plugins.sandbox-level.flash" from 0 to 1 or 1 in 64-bit Aurora 42 or Beta 41, I can reproduce the same Flash hang.

STR:
1. Install 64-bit Firefox. I am using Flash 18.0.0.232.
2. If 64-bit Silverlight is installed, set it to "Never Activate".
3. Play this Amazon Instant Video trailer:
http://www.amazon.com/gp/product/B00RSGGNBE/ref=dv_web_wtls_list_ovl_rs__wnzw?playertype=flash

4. Once the video starts playing, press the Back button or close the tab.

RESULT:
Firefox hangs. If e10s is disabled, Firefox displays a "Warning: Unresponsive Plugin" error dialog after about ten seconds. If you click the dialog's "Stop Plugin" button, Firefox will start responding again.

If e10s is enabled, however, Firefox never displays the error dialog. The browser window stops painting and goes white.

Comment 1

[Chris Peterson [:cpeterson]]

I can still repro on Win 8.1 and 10 VMs. I think closing the Amazon tab is irrelevant. You need to navigate to a new page (whether you close the Amazon tab or not) and/or open a new tab to see the "A web page is causing Nightly to run slowly" info bar.

If I Options > Kill Plugin, then the browser recovers. But if I resize *VMware* window running on my OS X host OS, then the browser window resizes and hangs forever. I can't click the info bar's Options button.

Comment 2

[Jim Mathies [:jimm]]

(In reply to Chris Peterson [:cpeterson] from comment #1)
> I can still repro on Win 8.1 and 10 VMs. I think closing the Amazon tab is
> irrelevant. You need to navigate to a new page (whether you close the Amazon
> tab or not) and/or open a new tab to see the "A web page is causing Nightly
> to run slowly" info bar.
> 
> If I Options > Kill Plugin, then the browser recovers. But if I resize
> *VMware* window running on my OS X host OS, then the browser window resizes
> and hangs forever. I can't click the info bar's Options button.

Might have something to do with using a vm, those tend to emulate single core cpus.

Comment 3

[Jeromie Clark]

This showed up on my radar because it got added to Chris's master hang bug that I'm tracking.  Just wanted to add some background.

My understanding is that our 32-bit sandbox is not enabled on x64 Firefox, and that Firefox is implementing an x64 sandbox for x64 Firefox.  

I'm not clear on what your dom.ipc.plugins.sandbox-level.flash flag does in the context of x64 Firefox, but I force-enabling our 32-bit sandbox on 64-bit Firefox should not be considered a valid test configuration.

If this is specific to 64-bit sandbox on 64-bit Firefox, I just want to make sure that you're aware that the 64-bit sandbox implementation is an internal Mozilla effort and that the bug should probably be routed to that engineering team for further investigation.

Comment 4

[Chris Peterson [:cpeterson]]

Thanks for the clarification, Jeromie. This hang is about Mozilla's 64-bit NPAPI sandbox, not Flash Protected Mode. I'll unlink this bug from the Flash hang tracking bug 1148076.

Comment 5

[Chris Peterson [:cpeterson]]

(In reply to Jim Mathies [:jimm] from comment #2)
> Might have something to do with using a vm, those tend to emulate single
> core cpus.

My Windows VMs are emulating two processor cores. Bug 1178438 is an example of a (32-bit) Firefox/VM hang caused by playing Silverlight content in a VM with just one processor.

Comment 6

[Jeromie Clark]

Out of curiosity, do you have the same issue on 1 or 4 cores?  We're investing a similar-sounding issue that we can only reproduce on 2 and 3 core VMs.  If that's the case, I'll get you a build with the fix to see if it resolves it for you.

Comment 7

[Chris Peterson [:cpeterson]]

I can reproduce both the Flash hang (STR in bug 1197943 comment 8) and this browser hang regardless of whether my VM has 1, 2, or 4 cores.

This bug is not limited to 64-bit Firefox. I see the same problem with 32-bit Firefox with dom.ipc.plugins.sandbox-level.flash pref = 1 and e10s enabled. If I switch away from the tab playing the video and back, the page goes black.

Comment 8

[Chris Peterson [:cpeterson]]

oops. Comment 7 should have said dom.ipc.plugins.sandbox-level.flash pref = 2

Comment 9

[Jeromie Clark]

This is Adobe 4044716

Comment 10

[Jim Mathies [:jimm]]

(In reply to Chris Peterson [:cpeterson] from comment #1)
> I can still repro on Win 8.1 and 10 VMs. I think closing the Amazon tab is
> irrelevant. You need to navigate to a new page (whether you close the Amazon
> tab or not) and/or open a new tab to see the "A web page is causing Nightly
> to run slowly" info bar.
> 
> If I Options > Kill Plugin, then the browser recovers. But if I resize
> *VMware* window running on my OS X host OS, then the browser window resizes
> and hangs forever. I can't click the info bar's Options button.

So aside from the resize oddity, you are seeing the slow plugin/script notification?

Comment 11

[Jim Mathies [:jimm]]

also, if you do get that, do you have any crash reports in about crashes?

Comment 12

[Jim Mathies [:jimm]]

(In reply to Chris Peterson [:cpeterson] from comment #7)
> I can reproduce both the Flash hang (STR in bug 1197943 comment 8) and this
> browser hang regardless of whether my VM has 1, 2, or 4 cores.
> 
> This bug is not limited to 64-bit Firefox. I see the same problem with
> 32-bit Firefox with dom.ipc.plugins.sandbox-level.flash pref = 1 and e10s
> enabled. If I switch away from the tab playing the video and back, the page
> goes black.

The black page is bug 1196539 and is unrelated to sandboxing.

I just tried to reproduce with:

1) adobe's sandbox disabled via the mms.cfg option
2) dom.ipc.plugins.sandbox-level.flash=2
3) 32-bit Nightly

I don't see any issues. Will try 64-bit next.

Comment 13

[Jim Mathies [:jimm]]

No luck with 64-bit. I'll try a vm next.

Comment 14

[Chris Peterson [:cpeterson]]

(In reply to Jim Mathies [:jimm] from comment #10)
> So aside from the resize oddity, you are seeing the slow plugin/script
> notification?

Here is my current STR. I just walked through the steps to ensure I got everything. I was able to repro five times in a row.

1. Run 64-bit Firefox.
2. Ensure dom.ipc.plugins.sandbox-level.flash=2
3. Enable e10s.
4. Restart Firefox.
5. In a non-maximized browser window, play video:
http://www.amazon.com/gp/product/B00RSGGNBE/ref=dv_web_wtls_list_ovl_rs__wnzw?playertype=flash
6. After video has played for a few seconds, enter http://wikipedia.org (or whatever) into the address bar and enter.
7. The tab icon shows the progress spinner.
8. Try switching tabs or dragging the browser window around the screen.

RESULT:
The chrome process hangs WITHOUT showing a slow page notification bar. Or if a slow page notification bar IS shown, try resizing the browser window by dragging the window corner or clicking the window's maximize/restore-down button. You do not need to resize the VM window of host OS.


(In reply to Jim Mathies [:jimm] from comment #11)
> also, if you do get that, do you have any crash reports in about crashes?

No crash reports.

Comment 15

[Jim Mathies [:jimm]]

So far I haven't had much luck getting amazon to load, but I do get a regular flash shutdown hang on pretty much every site that loads flash - 

https://crash-stats.mozilla.com/report/index/092a4a49-eb88-4940-bced-6d7ef2150831

Comment 16

[Jim Mathies [:jimm]]

(In reply to Jim Mathies [:jimm] from comment #15)
> So far I haven't had much luck getting amazon to load, but I do get a
> regular flash shutdown hang on pretty much every site that loads flash - 
> 
> https://crash-stats.mozilla.com/report/index/092a4a49-eb88-4940-bced-
> 6d7ef2150831

Hmm, this seems to have cured itself after a restart of the vm. I'm still not having any luck getting amazon video to load though, I see an error page that reads "We're experiencing a problem playing this video. Please try again. For further assistance, contact Amazon Customer Service."

Comment 17

[Jim Mathies [:jimm]]

recap thus far:

cpeterson has reproduced in both win 8.0/10.0 using vmware and 2 cores.

I've tried reproducing on windows 7.0 with no luck.

Locally I can not test using vmware+2cores+windows 8.1 since amazon will not play the video. I also am not able to test from the same ip on a lenovo yoga 3 running 8.1, and a surface pro 2 running windows 10.0. :/

We need more testers to confirm this issue and see if we can find a reliable way to repo.

Comment 18

[Liz Henry (:lizzard) (relman/hg->git project)]

Florin, can your team have a look and try to reproduce this usefully?  Thanks!

Comment 19

[Cornel Ionce [:noni] [Hubs QA]]

I've tried to test this but it seems that Amazon Instant video is region locked.
I'm getting this error: http://i.imgur.com/DMf2cpd.png

Tried with various US proxy configuration, without success.

Comment 20

[Bob Owen (:bobowen)]

I've reproduced this on Windows 8.1 VirtualBox and also on Windows 10 on my laptop.

Here's a crash report, which I think was generated when I killed the plugin process through the info bar option:
https://crash-stats.mozilla.com/report/index/5a8186f0-146d-437b-9624-3efe42150904

Looks like the plugin is trying to close a socket ... I'll investigate.

Comment 21

[Bob Owen (:bobowen)]

It looks like the socket that is being closed is a connection to some sort of CDN, which appears to be serving the content.

The initial closesocket does actually close the socket, but throws an exception and the Flash Player tries to close the socket again, which results in the hang.

Comment 22

[Bob Owen (:bobowen)]

OK it appears that it is the MITIGATION_STRICT_HANDLE_CHECKS process level mitigation that is breaking things here.

Flash seems to have an invalid handle somewhere, presumably to do with the socket being closed.

With the strict handle checks turned on an exception is thrown, which flash doesn't seem to be handling well.
The strict handle checks is only available for Win8+, so that explains why I never saw this.
I'm going to have to start manually testing on either Win8 or 10 or maybe both.

I think the invalid handle is due to other policy settings somehow, as if I just have the handle check then it seems fine.

Comment 23

[Bob Owen (:bobowen)]

I added back in the MITIGATION_STRICT_HANDLE_CHECKS, to try and reproduce this, but then I found that Amazon have already blocked flash by the looks of it.

Chris - did you see this hang on any other Flash sites?

Here's a build with the mitigation added back in, if you want to try and reproduce, thanks:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a241d0adc274

If you can't then we'll just close this.

Comment 24

[Chris Peterson [:cpeterson]]

I didn't see this Flash problem on any other websites, e.g. YouTube works correctly. Amazon Video no longer supports Flash and I don't have access to any other video services that use Flash DRM (assuming that was the problem).