Closed Bug 1198368 Opened 6 years ago Closed 6 years ago

[e10s] [ADBE 4044716] Playing Amazon Instant Video using Flash in NPAPI sandbox hangs the chrome process

Categories

(Core :: Plug-ins, defect)

All
Windows
defect
Not set
normal

Tracking

()

RESOLVED WORKSFORME
Tracking Status
e10s m8+ ---

People

(Reporter: blassey, Assigned: bobowen)

References

()

Details

(5 keywords, Whiteboard: [ADBE 4044716])

Separating out this bug to investigate why a plugin hang is able to hang the chrome process under e10s, but not with e10s disabled


+++ This bug was initially created as a clone of Bug #1197943 +++

[Tracking Requested - why for this release]:

This is a regression from bug 1185532, which enabled the NPAPI sandbox for Flash by default. If I change the "dom.ipc.plugins.sandbox-level.flash" from 0 to 1 or 1 in 64-bit Aurora 42 or Beta 41, I can reproduce the same Flash hang.

STR:
1. Install 64-bit Firefox. I am using Flash 18.0.0.232.
2. If 64-bit Silverlight is installed, set it to "Never Activate".
3. Play this Amazon Instant Video trailer:
http://www.amazon.com/gp/product/B00RSGGNBE/ref=dv_web_wtls_list_ovl_rs__wnzw?playertype=flash

4. Once the video starts playing, press the Back button or close the tab.

RESULT:
Firefox hangs. If e10s is disabled, Firefox displays a "Warning: Unresponsive Plugin" error dialog after about ten seconds. If you click the dialog's "Stop Plugin" button, Firefox will start responding again.

If e10s is enabled, however, Firefox never displays the error dialog. The browser window stops painting and goes white.
I can still repro on Win 8.1 and 10 VMs. I think closing the Amazon tab is irrelevant. You need to navigate to a new page (whether you close the Amazon tab or not) and/or open a new tab to see the "A web page is causing Nightly to run slowly" info bar.

If I Options > Kill Plugin, then the browser recovers. But if I resize *VMware* window running on my OS X host OS, then the browser window resizes and hangs forever. I can't click the info bar's Options button.
Summary: Playing Amazon Instant Video using 64-bit Flash in 64-bit Firefox hangs the chrome process → [e10s] Playing Amazon Instant Video using 64-bit Flash in 64-bit Firefox hangs the chrome process
(In reply to Chris Peterson [:cpeterson] from comment #1)
> I can still repro on Win 8.1 and 10 VMs. I think closing the Amazon tab is
> irrelevant. You need to navigate to a new page (whether you close the Amazon
> tab or not) and/or open a new tab to see the "A web page is causing Nightly
> to run slowly" info bar.
> 
> If I Options > Kill Plugin, then the browser recovers. But if I resize
> *VMware* window running on my OS X host OS, then the browser window resizes
> and hangs forever. I can't click the info bar's Options button.

Might have something to do with using a vm, those tend to emulate single core cpus.
This showed up on my radar because it got added to Chris's master hang bug that I'm tracking.  Just wanted to add some background.

My understanding is that our 32-bit sandbox is not enabled on x64 Firefox, and that Firefox is implementing an x64 sandbox for x64 Firefox.  

I'm not clear on what your dom.ipc.plugins.sandbox-level.flash flag does in the context of x64 Firefox, but I force-enabling our 32-bit sandbox on 64-bit Firefox should not be considered a valid test configuration.

If this is specific to 64-bit sandbox on 64-bit Firefox, I just want to make sure that you're aware that the 64-bit sandbox implementation is an internal Mozilla effort and that the bug should probably be routed to that engineering team for further investigation.
Thanks for the clarification, Jeromie. This hang is about Mozilla's 64-bit NPAPI sandbox, not Flash Protected Mode. I'll unlink this bug from the Flash hang tracking bug 1148076.
No longer blocks: flash-hangs
(In reply to Jim Mathies [:jimm] from comment #2)
> Might have something to do with using a vm, those tend to emulate single
> core cpus.

My Windows VMs are emulating two processor cores. Bug 1178438 is an example of a (32-bit) Firefox/VM hang caused by playing Silverlight content in a VM with just one processor.
See Also: → 1178438
Out of curiosity, do you have the same issue on 1 or 4 cores?  We're investing a similar-sounding issue that we can only reproduce on 2 and 3 core VMs.  If that's the case, I'll get you a build with the fix to see if it resolves it for you.
I can reproduce both the Flash hang (STR in bug 1197943 comment 8) and this browser hang regardless of whether my VM has 1, 2, or 4 cores.

This bug is not limited to 64-bit Firefox. I see the same problem with 32-bit Firefox with dom.ipc.plugins.sandbox-level.flash pref = 1 and e10s enabled. If I switch away from the tab playing the video and back, the page goes black.
Hardware: x86_64 → All
Summary: [e10s] Playing Amazon Instant Video using 64-bit Flash in 64-bit Firefox hangs the chrome process → [e10s] Playing Amazon Instant Video using Flash in NPAPI sandbox hangs the chrome process
oops. Comment 7 should have said dom.ipc.plugins.sandbox-level.flash pref = 2
Assignee: nobody → jmathies
See Also: → 1197940
This is Adobe 4044716
(In reply to Chris Peterson [:cpeterson] from comment #1)
> I can still repro on Win 8.1 and 10 VMs. I think closing the Amazon tab is
> irrelevant. You need to navigate to a new page (whether you close the Amazon
> tab or not) and/or open a new tab to see the "A web page is causing Nightly
> to run slowly" info bar.
> 
> If I Options > Kill Plugin, then the browser recovers. But if I resize
> *VMware* window running on my OS X host OS, then the browser window resizes
> and hangs forever. I can't click the info bar's Options button.

So aside from the resize oddity, you are seeing the slow plugin/script notification?
Flags: needinfo?(cpeterson)
also, if you do get that, do you have any crash reports in about crashes?
(In reply to Chris Peterson [:cpeterson] from comment #7)
> I can reproduce both the Flash hang (STR in bug 1197943 comment 8) and this
> browser hang regardless of whether my VM has 1, 2, or 4 cores.
> 
> This bug is not limited to 64-bit Firefox. I see the same problem with
> 32-bit Firefox with dom.ipc.plugins.sandbox-level.flash pref = 1 and e10s
> enabled. If I switch away from the tab playing the video and back, the page
> goes black.

The black page is bug 1196539 and is unrelated to sandboxing.

I just tried to reproduce with:

1) adobe's sandbox disabled via the mms.cfg option
2) dom.ipc.plugins.sandbox-level.flash=2
3) 32-bit Nightly

I don't see any issues. Will try 64-bit next.
No luck with 64-bit. I'll try a vm next.
(In reply to Jim Mathies [:jimm] from comment #10)
> So aside from the resize oddity, you are seeing the slow plugin/script
> notification?

Here is my current STR. I just walked through the steps to ensure I got everything. I was able to repro five times in a row.

1. Run 64-bit Firefox.
2. Ensure dom.ipc.plugins.sandbox-level.flash=2
3. Enable e10s.
4. Restart Firefox.
5. In a non-maximized browser window, play video:
http://www.amazon.com/gp/product/B00RSGGNBE/ref=dv_web_wtls_list_ovl_rs__wnzw?playertype=flash
6. After video has played for a few seconds, enter http://wikipedia.org (or whatever) into the address bar and enter.
7. The tab icon shows the progress spinner.
8. Try switching tabs or dragging the browser window around the screen.

RESULT:
The chrome process hangs WITHOUT showing a slow page notification bar. Or if a slow page notification bar IS shown, try resizing the browser window by dragging the window corner or clicking the window's maximize/restore-down button. You do not need to resize the VM window of host OS.


(In reply to Jim Mathies [:jimm] from comment #11)
> also, if you do get that, do you have any crash reports in about crashes?

No crash reports.
Flags: needinfo?(cpeterson)
Summary: [e10s] Playing Amazon Instant Video using Flash in NPAPI sandbox hangs the chrome process → [e10s] [ADBE 4044716] Playing Amazon Instant Video using Flash in NPAPI sandbox hangs the chrome process
Whiteboard: [ADBE 4044716]
So far I haven't had much luck getting amazon to load, but I do get a regular flash shutdown hang on pretty much every site that loads flash - 

https://crash-stats.mozilla.com/report/index/092a4a49-eb88-4940-bced-6d7ef2150831
(In reply to Jim Mathies [:jimm] from comment #15)
> So far I haven't had much luck getting amazon to load, but I do get a
> regular flash shutdown hang on pretty much every site that loads flash - 
> 
> https://crash-stats.mozilla.com/report/index/092a4a49-eb88-4940-bced-
> 6d7ef2150831

Hmm, this seems to have cured itself after a restart of the vm. I'm still not having any luck getting amazon video to load though, I see an error page that reads "We're experiencing a problem playing this video. Please try again. For further assistance, contact Amazon Customer Service."
recap thus far:

cpeterson has reproduced in both win 8.0/10.0 using vmware and 2 cores.

I've tried reproducing on windows 7.0 with no luck.

Locally I can not test using vmware+2cores+windows 8.1 since amazon will not play the video. I also am not able to test from the same ip on a lenovo yoga 3 running 8.1, and a surface pro 2 running windows 10.0. :/

We need more testers to confirm this issue and see if we can find a reliable way to repo.
Keywords: qawanted
Florin, can your team have a look and try to reproduce this usefully?  Thanks!
Flags: needinfo?(florin.mezei)
I've tried to test this but it seems that Amazon Instant video is region locked.
I'm getting this error: http://i.imgur.com/DMf2cpd.png

Tried with various US proxy configuration, without success.
Flags: needinfo?(florin.mezei)
I've reproduced this on Windows 8.1 VirtualBox and also on Windows 10 on my laptop.

Here's a crash report, which I think was generated when I killed the plugin process through the info bar option:
https://crash-stats.mozilla.com/report/index/5a8186f0-146d-437b-9624-3efe42150904

Looks like the plugin is trying to close a socket ... I'll investigate.
Assignee: jmathies → bobowen.code
It looks like the socket that is being closed is a connection to some sort of CDN, which appears to be serving the content.

The initial closesocket does actually close the socket, but throws an exception and the Flash Player tries to close the socket again, which results in the hang.
OK it appears that it is the MITIGATION_STRICT_HANDLE_CHECKS process level mitigation that is breaking things here.

Flash seems to have an invalid handle somewhere, presumably to do with the socket being closed.

With the strict handle checks turned on an exception is thrown, which flash doesn't seem to be handling well.
The strict handle checks is only available for Win8+, so that explains why I never saw this.
I'm going to have to start manually testing on either Win8 or 10 or maybe both.

I think the invalid handle is due to other policy settings somehow, as if I just have the handle check then it seems fine.
I added back in the MITIGATION_STRICT_HANDLE_CHECKS, to try and reproduce this, but then I found that Amazon have already blocked flash by the looks of it.

Chris - did you see this hang on any other Flash sites?

Here's a build with the mitigation added back in, if you want to try and reproduce, thanks:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a241d0adc274

If you can't then we'll just close this.
Flags: needinfo?(cpeterson)
I didn't see this Flash problem on any other websites, e.g. YouTube works correctly. Amazon Video no longer supports Flash and I don't have access to any other video services that use Flash DRM (assuming that was the problem).
Status: NEW → RESOLVED
Closed: 6 years ago
Flags: needinfo?(cpeterson)
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.