Flash plugin process hangs when playing Amazon Instant Video with NPAPI sandbox enabled (e.g. 64-bit Firefox)

RESOLVED FIXED in Firefox 41

Status

()

defect
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: cpeterson, Assigned: bobowen)

Tracking

(5 keywords)

unspecified
mozilla43
x86_64
Windows
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(e10s-, firefox40 unaffected, firefox41 fixed, firefox42+ fixed, firefox43+ fixed)

Details

(crash signature, )

Attachments

(1 attachment)

[Tracking Requested - why for this release]:

This is a regression from bug 1185532, which enabled the NPAPI sandbox for Flash by default. If I change the "dom.ipc.plugins.sandbox-level.flash" from 0 to 1 or 1 in 64-bit Aurora 42 or Beta 41, I can reproduce the same Flash hang.

STR:
1. Install 64-bit Firefox. I am using Flash 18.0.0.232.
2. If 64-bit Silverlight is installed, set it to "Never Activate".
3. Play this Amazon Instant Video trailer:
http://www.amazon.com/gp/product/B00RSGGNBE/ref=dv_web_wtls_list_ovl_rs__wnzw?playertype=flash

4. Once the video starts playing, press the Back button or close the tab.

RESULT:
Firefox hangs. If e10s is disabled, Firefox displays a "Warning: Unresponsive Plugin" error dialog after about ten seconds. If you click the dialog's "Stop Plugin" button, Firefox will start responding again.

If e10s is enabled, however, Firefox never displays the error dialog. The browser window stops painting and goes white.
@ Brad, Bob is on PTO. Who can investigate this Flash hang (or back out bug 1185532)?

Aurora 42 and Beta 41 are affected if I change the "dom.ipc.plugins.sandbox-level.flash" from 0 to 1 or 1 in 64-bit Aurora 42 or Beta 41.
Flags: needinfo?(blassey.bugs)
Keywords: 64bit
Block Win64 tracking bug 471090.
(In reply to Chris Peterson [:cpeterson] from comment #1)
> @ Brad, Bob is on PTO. Who can investigate this Flash hang (or back out bug
> 1185532)?
I don't think we should back out given that lack of Protected Mode in 64bit Flash. The alternative would be to disable flash.

Is there any reason this can't wait until Bob returns from PTO?
Flags: needinfo?(blassey.bugs)
Brad, this bug hangs the entire browser when e10s is enabled, which is the default in Nightly 43 and Aurora 42. 64-bit Firefox doesn't support Silverlight, so people will need to use Flash to watch Amazon Instant Videos, which will now hang their browser. Is hanging the browser preferable to having no sandbox in the pre-release channels?

e10s is not enabled in Beta 42, so we could uplift bug 1185532 to Beta, but the sandbox would not have had any testing in Nightly or Aurora.
Flags: needinfo?(blassey.bugs)
See Also: → 1197940
(In reply to Chris Peterson [:cpeterson] from comment #4)
> Brad, this bug hangs the entire browser when e10s is enabled, which is the
> default in Nightly 43 and Aurora 42. 64-bit Firefox doesn't support
> Silverlight, so people will need to use Flash to watch Amazon Instant
> Videos, which will now hang their browser. Is hanging the browser preferable
> to having no sandbox in the pre-release channels?
I think the choice we're looking at here is keeping the sandbox or blacklisting flash for 64bit.
 
> e10s is not enabled in Beta 42, so we could uplift bug 1185532 to Beta, but
> the sandbox would not have had any testing in Nightly or Aurora.
The fact that this hang is effecting the chrome process is a separate problem that we need to look at. I'm going to clone this bug for that.
Flags: needinfo?(blassey.bugs)
Does the browser recover after the 45-second chrome hang time out?
hmm, I can't reproducing using the str in comment 1 on win7.
I can still repro on Win 8.1 and 10 VMs. I think closing the Amazon tab is irrelevant. You need to navigate to a new page (whether you close the Amazon tab or not) for the browser to show the "Warning: Unresponsive Plugin" error dialog.

(I will move discussion of the e10s browser hangs to bug 1198368.)
Summary: 64-bit Firefox hangs when playing Amazon Instant Video using 64-bit Flash → 64-bit Flash plugin process hangs when playing Amazon Instant Video
Like e10s hang bug 1198368, this bug is not limited to 64-bit Firefox. I see the same problem with 32-bit Firefox with dom.ipc.plugins.sandbox-level.flash pref = 2.
Duplicate of this bug: 1197940
I see the firefox.exe and plugin-container.exe are "waiting to finish network I/O" in "Analyze wait chain" of Task Manager on Win8.1, Fx41b4 (no e10s), dom.ipc.plugins.sandbox-level.flash = 2 when hang with single tab opened http://www.bbc.co.uk/programmes/p02zfwz9.

And after a long period of time, I see the plugin-hang-ui and plugin crashed ui, tried twice.
https://crash-stats.mozilla.com/report/index/69bec5db-656e-4886-b12c-276282150827
https://crash-stats.mozilla.com/report/index/4622f8c8-9f41-4d32-8481-232a32150827


Seem the also happen for dom.ipc.plugins.sandbox-level.flash = 1
https://crash-stats.mozilla.com/report/index/c7c1ea50-6cbe-49a8-bbc7-b05dd2150827


Note: Looks the hang will happen at the video play end.
Crash Signature: [@ hang | RtlpWaitOnCriticalSection | mswsock.dll@0x4d04f ]
Tracking this. 
Chris should we change the title to reflect that it isn't just a 64-bit issue?
YF are you still working on this or do we still need to find an owner for this bug?
Flags: needinfo?(yfdyh000)
Flags: needinfo?(cpeterson)
I think this signature might correlate to this problem - 

https://crash-stats.mozilla.com/report/list?product=Firefox&range_value=7&range_unit=days&date=2015-09-01&signature=WaitForMultipleObjectsEx&version=Firefox%3A43.0a1

1) all reports appear to be from windows although we seem to be missing os version info
2) all appear to be 64bit
3) of the pages I could confirm, all point to video related pages
4) a mixture of graphics cards
5) uptime varies, not a startup crash
6) the signature has been around for a bit but it appears to have regressed in a big way 8-24-15

Hard to confirm though since 64-bit crash data appears to be messed up.
also, about 90% are nightly, the rest on aurora.
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #12)
> Tracking this. 
> Chris should we change the title to reflect that it isn't just a 64-bit
> issue?

We don't need to track this bug for 41 because we won't ship 64-bit Firefox until 42 or later. See bug 1181014.

I updated the bug title to mention the NPAPI sandbox instead of 64-bit. The NPAPI sandbox is enabled by default in 64-bit Firefox, which is why bug originally mentioned on 64-bit.

I also moved the bug from the Plug-Ins component to Security: Process Sandboxing, since this is a bug in our NPAPI sandbox, not Flash itself.
Component: Plug-ins → Security: Process Sandboxing
Flags: needinfo?(cpeterson)
Summary: 64-bit Flash plugin process hangs when playing Amazon Instant Video → Flash plugin process hangs when playing Amazon Instant Video with NPAPI sandbox enabled (e.g. 64-bit Firefox)
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #12)
> Tracking this. 
> Chris should we change the title to reflect that it isn't just a 64-bit
> issue?
> YF are you still working on this or do we still need to find an owner for
> this bug?

We're not using our own sandbox on 32-bit as we have Flash Player's own protected mode sandbox.

I'll look into this when I'm back on Friday.
Assignee: nobody → bobowen.code
(In reply to Jim Mathies [:jimm] from comment #15)
> also, about 90% are nightly, the rest on aurora.

> 6) the signature has been around for a bit but it appears to have regressed
> in a big way 8-24-15

I think we should back bug 1185532 out from beta. The landings there correlate well with this hang signature.
Crash Signature: [@ hang | RtlpWaitOnCriticalSection | mswsock.dll@0x4d04f ] → [@ hang | RtlpWaitOnCriticalSection | mswsock.dll@0x4d04f ] [@ WaitForMultipleObjectsEx ]
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #12)
> Tracking this. 
> Chris should we change the title to reflect that it isn't just a 64-bit
> issue?
> YF are you still working on this or do we still need to find an owner for
> this bug?

No, I not works on this bug, just report it.
Flags: needinfo?(yfdyh000)
A lot of these appear to be in destroy calls, but not all. Here's a break down of all plugin hang stacks for 42 -

http://www.mathies.com/mozilla/plugin-hang-report.html

search for:

os: Windows NT, 10.0.10240
plugin: NPSWF64_18_0_0_232.dll
monitor: HangMonitor
Ah, I've been commenting against bug 1198368 for this issue, which I think was raised just to look into why this hangs the chrome process as well, when e10s is enabled.

So see bug 1198368 comment 22 for some more detail.
Bug 1197943: Turn off MITIGATION_STRICT_HANDLE_CHECKS for NPAPI process sandbox for causing hangs. r?aklotz
Attachment #8657235 - Flags: review?(aklotz)
Duplicate of this bug: 1202235
Keywords: crash
Comment on attachment 8657235 [details]
MozReview Request: Bug 1197943: Turn off MITIGATION_STRICT_HANDLE_CHECKS for NPAPI process sandbox for causing hangs. r?aklotz

https://reviewboard.mozilla.org/r/18345/#review16669
Attachment #8657235 - Flags: review?(aklotz) → review+
https://hg.mozilla.org/integration/mozilla-inbound/rev/63d9b41521e90bb8cef3c2f86c05a274436d9384
Bug 1197943: Turn off MITIGATION_STRICT_HANDLE_CHECKS for NPAPI process sandbox for causing hangs. r=aklotz
https://hg.mozilla.org/mozilla-central/rev/63d9b41521e9
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla43
Comment on attachment 8657235 [details]
MozReview Request: Bug 1197943: Turn off MITIGATION_STRICT_HANDLE_CHECKS for NPAPI process sandbox for causing hangs. r?aklotz

Approval Request Comment
[Feature/regressing bug #]:
Bug 1185532 - turned on the NPAPI sandbox for Flash on 64-bit by default.

[User impact if declined]:
Users testing 64-bit Firefox on Win8+ will continue to get hangs if they try to navigate while playing Flash.
There are some other things that trigger the same problem as well.

[Describe test coverage new/current, TreeHerder]:
No Flash coverage in tree, but tested original problem and the one from bug 1197940 manually.

[Risks and why]:
Low - this is a very simple change that just removes one of the sandbox restrictions.
This restriction causes exceptions to be thrown on a bad handle reference, which was then causing flash to hang.

[String/UUID change made/needed]:
None
Attachment #8657235 - Flags: approval-mozilla-beta?
Attachment #8657235 - Flags: approval-mozilla-aurora?
Comment on attachment 8657235 [details]
MozReview Request: Bug 1197943: Turn off MITIGATION_STRICT_HANDLE_CHECKS for NPAPI process sandbox for causing hangs. r?aklotz

Taking it for aurora as we want to improve the fx windows 64 bit situation. However, I don't think Ritu will take it in beta as it is really late in the cycle.
Attachment #8657235 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
Hmm, I think we can't really ship plugin sandboxing without this fix, which means we would not be able to ship Win64 builds for 41 without taking this in beta.

Am I right that this should really only affect Win64 at this point?
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #29)
> Hmm, I think we can't really ship plugin sandboxing without this fix, which
> means we would not be able to ship Win64 builds for 41 without taking this
> in beta.
> 
> Am I right that this should really only affect Win64 at this point?

Yes, this only affects 64-bit Windows as we only have the NPAPI sandbox turned on for 64-bit flash.

I don't think we are shipping Win64 for 41 anyway.
However, it would be good to get this in for the rest of Beta 41.
(In reply to Bob Owen (:bobowen) from comment #30)
> I don't think we are shipping Win64 for 41 anyway.

According to bug 1181014 we are shipping Win64 only in 42 (though initially it was indeed 41).
Comment on attachment 8657235 [details]
MozReview Request: Bug 1197943: Turn off MITIGATION_STRICT_HANDLE_CHECKS for NPAPI process sandbox for causing hangs. r?aklotz

Based on what I have been told this only impacts win64 builds. 41 does not offically support win64 but we do want our win64 story to get solid and therefore taking this patch for Beta41.
Attachment #8657235 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
You need to log in before you can comment on or make changes to this bug.