1082670 - [adbe 3839666] Flash plugin crashes/hangs every time in Nightly, after bug 1061335

Status: VERIFIED FIXED

(Core Graveyard :: Plug-ins, defect)

Description

[Gary [:streetwolf52]]

Regression results:

20141013192600
https://hg.mozilla.org/integration/mozilla-inbound/rev/e0c4c804b279  - Good

20141013195601
https://hg.mozilla.org/integration/mozilla-inbound/rev/ea2ea74d0e33  - Bad

Comment 1

[Ahmad Saleem]

CSET: https://hg.mozilla.org/mozilla-central/rev/54217864bae9

This video playing fine - http://www.dailymotion.com/video/x27x16j with Adobe Flash 15.0.0.189 Stable. 

DLL File: NPSWF32_15_0_0_189.dll

Windows 8.1 Update 1 64-bit but 32-bit Nightly.

Comment 2

[Jim Jeffery not reading bug-mail 1/2/11]

Confirmed  - setting to NEW

Comment 3

[Jim Jeffery not reading bug-mail 1/2/11]

Flash plug-in is crashing, but getting the 'generic' Flash crash report we've been seeing for ages now:

https://crash-stats.mozilla.com/report/index/78d98011-1762-458f-844a-f6bc42141014

Comment 4

[Gary [:streetwolf52]]

I'm not getting any crashes, just no video on sites.

Comment 5

[IU]

WFM with Flash Player 15.0.0.189.  Got specific examples?

Comment 6

[Gary [:streetwolf52]]

Any video at cnn.com

Comment 7

[Jim Jeffery not reading bug-mail 1/2/11]

(In reply to IU from comment #5)
> WFM with Flash Player 15.0.0.189.  Got specific examples?


Well, I tested on CNN's web-site, and was getting flash-player crashes.

A quick test just now on Youtube, flashplayer does not crash.  So maybe its related to CNN's player... I believe they use some extra coding on their video.  Octoshape add-in for Flashplayer is what they use as I recall.

Comment 8

[IU]

CNN WFM.  Try updating your Flash Player to 15.0.0.189, if you haven't already.

And in case it matters, I'm using Windows 8.1.

YouTube is no longer Flash by default (unless you've done something to force it), so make sure it's actually a Flash video you're viewing.

Comment 9

[Gary [:streetwolf52]]

YouTube uses HTML5 which is not affected.  Flash doesn't play on other sites like weather.com

Comment 10

[Gary [:streetwolf52]]

The latest beta Flash is 15.0.0.199.  Give that a try.

Comment 11

[Alice0775 White]

(In reply to Jim Jeffery not reading bug-mail 1/2/11 from comment #7)
> (In reply to IU from comment #5)
> > WFM with Flash Player 15.0.0.189.  Got specific examples?
> 
> 
> Well, I tested on CNN's web-site, and was getting flash-player crashes.
> 
> A quick test just now on Youtube, flashplayer does not crash.  So maybe its
> related to CNN's player... I believe they use some extra coding on their
> video.  Octoshape add-in for Flashplayer is what they use as I recall.

I do not think that the problem is not limited in CNN.

 
http://www.adobe.com/software/flash/about/ site should show Flash Version.
However, Flash crashes or just empty or gray rectangle.

bp-7c6d6c00-b9c5-416d-8bec-9b88e2141014
bp-eea5dae6-91dc-4b30-8e02-585642141014

Comment 12

[Alice0775 White]

sorry, my English translation error , 

I think that the problem is not limited in CNN.

Comment 13

[Jim Jeffery not reading bug-mail 1/2/11]

(In reply to IU from comment #8)
> CNN WFM.  Try updating your Flash Player to 15.0.0.189, if you haven't
> already.
> 
> And in case it matters, I'm using Windows 8.1.
> 
> YouTube is no longer Flash by default (unless you've done something to force
> it), so make sure it's actually a Flash video you're viewing.

I'm already running 15.0.0..189 as of today, but... I'm on win7 x64.  This is getting confusing quickly...

NO crash on Adobe site from comment #11 here.

Youtube is NOT all HTML5 all the time, lately I've been getting a lot of fall-backs to Flash for some reason.

Comment 14

[Gary [:streetwolf52]]

As Alice mentioned over at MozillaZine setting environment variable MOZ_DISABLE_OOP_PLUGINS=1 makes Flash work for me.

Comment 15

[Benjamin Smedberg]

I can't reproduce this either on http://www.adobe.com/software/flash/about/ or the dailymotion URL, using a win32 build on win7x64.

Comment 16

[IU]

Does it work if you disable protected mode?
http://kb.mozillazine.org/Flash#Disabling_Protected_Mode_in_Flash_11.3

Comment 17

[Gary [:streetwolf52]]

(In reply to IU from comment #16)
> Does it work if you disable protected mode?
> http://kb.mozillazine.org/Flash#Disabling_Protected_Mode_in_Flash_11.3

Yes, Flash works fine when disabling protected mode.  So there are two ways around the problem so far.

Comment 18

[Benjamin Smedberg]

Perhaps obviously, those workarounds are not acceptable in the general case. We need to understand why this is actually failing.

Comment 19

[Gary [:streetwolf52]]

If there is anything I can do to help resolve this let me know. Keep in mind I am just an end-user.  Do you ever use something like TeamViewer to help with a problem?

Comment 20

[(Away)]

I get a crash in FlashPlayerPlugin_15_0_0_199.exe (but not plugin-container, and not firefox) on Win7 SP1 at http://edition.cnn.com/video.

Near the crash site I see the standard sequence of tests for "is there a valid PE image at this address", but there's nothing at that address (it's reserved but not committed).

Odd that a change in FF would cause this in Flash's process. Are they mapping FF binaries into their exe's memory? Maybe that logic is getting messed up by the new CRT files or the new version linker flags on the other binaries.

bsmedberg do you have a way to reach somebody at Adobe?

Comment 21

[Benjamin Smedberg]

Yes, I'll ping Adobe.

Comment 22

[David Weir (satdav)]

This has happened on today build and above I can give you my build ids

Comment 23

[David Weir (satdav)]

Ps happens on Facebook games from today 

so I will close my bug I opened and mark duplicate of this

Comment 25

[(Away)]

Interesting. When I go here: http://webhtb.sourceforge.net/video_demo.html (per bug 1083504), then I get my postmortem debugger for the crash in FlashPlayerPlugin_15_0_0_199.exe, but then later I see a frowny-face for the crash in plugin-container.exe, at the same F1398665248 symbol that others are seeing: bp-fd904735-3f90-4c01-887b-d92522141016.

I guess if one didn't have a postmortem debugger, only the plugin-container.exe crash would come up.

Comment 26

[(Away)]

The plugin-container crash is an int 3, so I assume that's just a panic resulting from the original crash in FlashPlayerPlugin.

Comment 27

[Trevor Rowbotham [:rowbot]]

This crash stat may be of some interest since it is slightly different than the more generic one that everyone has been posting, though I also have plenty of those generic ones as well. Hope it helps.

https://crash-stats.mozilla.com/report/index/f4828673-045b-453e-ad24-4c8152141016

Comment 28

[Jeromie Clark]

We're looking into it.  This is Adobe 3839666

Comment 31

[Gary [:streetwolf52]]

(In reply to Jeromie Clark from comment #28)
> We're looking into it.  This is Adobe 3839666

Can we get a link to the Adobe bug report filed?  I can't seem to find it when I do a search on Adobe's bug site.

Comment 33

[Achwaq Khalid]

The flash plugin just crashes on here http://www.piliapp.com/facebook-symbols/ !!

Here are the related signatures:

be0aa937-44ea-4e5a-8fae-10509aeb376f	10/17/2014	12:03 AM
bp-0a4377eb-6742-434a-8782-865dd2141016	10/16/2014	10:07 PM
6d64c946-2697-4754-b2a3-3b4363572b6e	10/16/2014	4:26 PM
03c794e5-57ee-4c34-9776-53ad50e47203	10/16/2014	4:12 PM
6738a4c8-d411-4147-9ee8-836600aa0d08	10/16/2014	3:54 PM
13d4db8c-85f2-4d15-a553-9aded82bd152	10/16/2014	12:32 AM
dccd08e5-27aa-47d2-b455-88e94f7a77a5	10/16/2014	12:26 AM
3eab6124-cbde-4cc1-88b5-e818df38b746	10/15/2014	9:49 PM
86c106ed-6e22-4783-9b9d-4a6f22c77e73	10/15/2014	7:32 PM
0a1f3334-1721-4117-bec5-8f51e3633bab	10/15/2014	7:31 PM
f95770d5-6263-44e6-abe8-0fa8c9e26626	10/15/2014	7:21 PM
0c6877a7-d544-4f49-864d-9d877248adc1	10/15/2014	6:42 PM

Comment 34

[Wilhelm]

When I use the 64bit version of Firefox 36.0A1, Flashplayer works perfectly, it seems that the 32-bit versions of firefox do not work with flashplayerd!

Comment 35

[alex_mayorga]

(In reply to Gary [:streetwolf] from comment #31)
> (In reply to Jeromie Clark from comment #28)
> > We're looking into it.  This is Adobe 3839666
> 
> Can we get a link to the Adobe bug report filed?  I can't seem to find it
> when I do a search on Adobe's bug site.

¡Hola Mr. Clark!

Any progress on this bug from Adobe?

¡Gracias!
Alex

Comment 37

[Lily Kim]

(In reply to Gary [:streetwolf] from comment #31)
> (In reply to Jeromie Clark from comment #28)
> > We're looking into it.  This is Adobe 3839666
> 
> Can we get a link to the Adobe bug report filed?  I can't seem to find it
> when I do a search on Adobe's bug site.

Me neither. Is it possible the bug report is private?

Comment 39

[Jeromie Clark]

We have separate external and internal bug repositories.  I can create an external bug, but it will basically show as Open until it's Closed, which doesn't really add any value for you.  I provide you guys with the internal bug numbers primarily just to speed communication between the two engineering groups.

Comment 40

[Jeffrey]

I am not seeing videos when I try access videos from YouTube or http://www.chicagotribune.com/ for example.
I have Flash 15.0.0152.
I also tried Flash 15.0.0.189 and got the same results, black screen.

FF 34.1

Comment 41

[Loic]

(In reply to jlerner10 from comment #40)
> I am not seeing videos when I try access videos from YouTube or
> http://www.chicagotribune.com/ for example.
> I have Flash 15.0.0152.
> I also tried Flash 15.0.0.189 and got the same results, black screen.
> 
> FF 34.1

This bug is about FF36, not prior versions.

Comment 42

[Jeromie Clark]

Nope.  This is not a dire emergency from our perspective, as you do have the option to roll back to the old project files.  I can tell you that updating Flash Player to VS2013 to resolve this is not an achievable outcome in the short-term.

This issue is in the queue and has been assigned to an engineer for investigation.  When we have finished our analysis and can provide you with useful feedback, we'll update this bug.

Comment 43

[IU]

At its root, I believe this issue has something to do with the interaction between Flash Player protected mode and Gecko IPC.  That disabling one or the other resolves the issue is evidence of this.

I even have protected mode disabled permanently on two systems because it causes stalls and horribly jerky playback of lots of videos.  Granted, it could be IPC that's at fault, but Mozilla not longer supports having IPC disabled.

Sure there are risks, but I have click-to-play enabled for Flash by default.

It could be a Flash problem or it could simply be a side-effect of an IPC problem, so (rather than just waiting on Adobe) it would be prudent of Mozilla to simultaneously investigate Gecko IPC behavior.

Comment 45

[timbugzilla]

64bit Nightly seems OK for me. 32bit has constant hangs.

Comment 46

[Jeromie Clark]

One of the interesting tidbits of feedback that I've been seeing lately for users in the Flash forums is that disabling PluginContainer (via the OOP-related environmental variables) resolves a lot of the stability problems that people associated with Flash Player Protected Mode.  I also noticed that when running in ProtectedMode, a crash in Flash Player doesn't take down the tab or browser when it crashes.  It might be a low-cost win that could be worth exploring independent of the current bug.

Comment 47

[Wilhelm]

Putting  ProtectedMode=0
in mms.cfg  C:\Windows\SysWOW64\Macromed\Flash
works.

Comment 48

[Bur]

Setting MOZ_DISABLE_OOP_PLUGINS=1 in system environment works for me as well. Otherwise Flash player crashes.

Win 7 x64
Nightly 36 32bit

Comment 49

[Benjamin Edwards]

Confirming that setting ProtectedMode=0 in mms.cfg works for beta build 15.0.0.207 as well.

Comment 52

[Ajay Gupta]

I tried one of the fix provided in https://bugzilla.mozilla.org/show_bug.cgi?id=1082670 and the flash seems to be working fine now in the latest nightly.

Comment 53

[Ajay Gupta]

Ignore the previous comment. It was meant to be attached to bug 810629.

Comment 54

[Benjamin Smedberg]

We will not be disabling plugin-container, and in fact it will soon be the only available option, since it is required to support multi-process content.

Comment 55

[IU]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #54)
> We will not be disabling plugin-container, and in fact it will soon be the
> only available option, since it is required to support multi-process content.

Could you guys at least troubleshoot/inspect Gecko IPC with respect to this bug?  I strongly believe this is actually an IPC bug and not a Flash Player protected mode bug.

Comment 56

[Benjamin Smedberg]

I managed to narrow this down: the crash appears to be related to the crash reporting code that Firefox injects into the FlashPlayerPlugin.exe processes (bug 769048 has history). Likely due to the CRT upgrade, but we'll need to debug further to figure out exactly what's going on.

Comment 57

[Jeromie Clark]

Cool, let us know if there's anything we can do to help.  Happy to provide symbols, etc.

Comment 59

[(Away)]

The crash is indeed happening in the crash reporting DLL:

breakpadinjector!_ValidateImageBase
breakpadinjector!_IsNonwritableInCurrentImage
breakpadinjector!_cinit
breakpadinjector!_CRT_INIT
breakpadinjector!Start

The relevant piece of _cinit is:
        if (_FPinit != NULL &&
            _IsNonwritableInCurrentImage((PBYTE)&_FPinit))
        {
            (*_FPinit)(initFloatingPrecision);
        }

That code initializes the CRT's floating-point support, if the program expects to use it. The breakpadinjector.dll from VS2010 had a null _FPinit, so we never called _IsNonwritableInCurrentImage. With VS2013 we have a non-null _FPinit routine. (I don't know why -- maybe some SDK or CRT header used by injector.cpp added a float variable somewhere.)

_ValidateImageBase crashes because the memory at the start of breakpadinjector.dll is decommitted. I suspect that our FinalizeSections [1] didn't map the PE header because it's not part of any section.

Interestingly, _ValidateImageBase protects itself with a try/catch, but because the handler code is not part of a bona-fide MEM_IMAGE, the NT exception dispatcher says no-go.

I propose that we add an extra VirtualAllocEx to commit breakpadinjector's image header.

[1] http://dxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/LoadLibraryRemote.cpp#112

Comment 60

[(Away)]

Attachment: Commit the header

Comment 61

[(Away)]

Comment on attachment 8508371 [details] [diff] [review]
Commit the header

Er, it would help if I followed it up with a WriteProcessMemory :)

Comment 62

[(Away)]

Attachment: Copy the breakpadinjector image header

Now with actually initialized data.

0:011> db .-159a
00ec0000  4d 5a 90 00 03 00 00 00-04 00 00 00 ff ff 00 00  MZ..............
00ec0010  b8 00 00 00 00 00 00 00-40 00 00 00 00 00 00 00  ........@.......
00ec0020  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
00ec0030  00 00 00 00 00 00 00 00-00 00 00 00 f8 00 00 00  ................
00ec0040  0e 1f ba 0e 00 b4 09 cd-21 b8 01 4c cd 21 54 68  ........!..L.!Th
00ec0050  69 73 20 70 72 6f 67 72-61 6d 20 63 61 6e 6e 6f  is program canno
00ec0060  74 20 62 65 20 72 75 6e-20 69 6e 20 44 4f 53 20  t be run in DOS 
00ec0070  6d 6f 64 65 2e 0d 0d 0a-24 00 00 00 00 00 00 00  mode....$.......

Comment 63

[(Away)]

Comment on attachment 8508378 [details] [diff] [review]
Copy the breakpadinjector image header

Review of attachment 8508378 [details] [diff] [review]:
-----------------------------------------------------------------

::: toolkit/crashreporter/injector/injector.cpp
@@ +23,5 @@
>  
>  extern "C"
>  __declspec(dllexport) DWORD Start(void* context)
>  {
> +__debugbreak();

Bah, forgot to qref this. I'll remove before submitting. Same for the trailing whitespace in the previous file.

Comment 66

[Ed Morley [:emorley]]

(Making summary more keyword rich, to try and avoid dupes).

Comment 68

[(Away)]

 https://hg.mozilla.org/integration/mozilla-inbound/rev/ac44aef9c4d4

Comment 69

[Jeromie Clark]

Based on the feedback, I'm going to close the bug on our side.  Let me know if you need anything further from Adobe.

Comment 70

[(Away)]

Sounds good. Thanks for keeping an eye on this, Jeromie.

Comment 71

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/ac44aef9c4d4

Comment 72

[Achwaq Khalid]

Firefox 36.0a1 is still affected...

Comment 73

[(No longer employed by Mozilla) Aaron Klotz]

(In reply to Achwaq Khalid from comment #72)
> Firefox 36.0a1 is still affected...

Are you running e10s? I had to fix a plugin-related crash for that in bug 1087410.

Comment 74

[Achwaq Khalid]

(In reply to Aaron Klotz [:aklotz] from comment #73)
> (In reply to Achwaq Khalid from comment #72)
> > Firefox 36.0a1 is still affected...
> 
> Are you running e10s? I had to fix a plugin-related crash for that in bug
> 1087410.

Yes Aaron, I have E10S enabled

Comment 76

[Suburp]

Are you sure this has been solved?

https://crash-stats.mozilla.com/report/index/4a221aa4-146b-4558-a467-9766a2141202

Still occurs in Ff33 on Dec 02:
dgapi.dll 	dgapi.dll@0xde824
dgapi.dll 	dgapi.dll@0x33fab
Date Processed	2014-12-02 13:38:47.256006
Process Type 	plugin Shockwave Flash Version:15.0.0.239 Filename:NPSWF32_15_0_0_239.dll

Comment 77

[Suburp]

Same in V32
https://crash-stats.mozilla.com/report/index/9810fca5-7fbb-4cc0-b4dc-e92f12141003
https://crash-stats.mozilla.com/report/index/4f062917-612a-4d09-90ff-1c8042141003
https://crash-stats.mozilla.com/report/index/e60de6d3-9278-4f73-b8e6-094422141017
and seems to do the same in V36, too.

Comment 78

[Loic]

(In reply to Suburp from comment #76)
> Are you sure this has been solved?
> 
> https://crash-stats.mozilla.com/report/index/4a221aa4-146b-4558-a467-
> 9766a2141202
> 
> Still occurs in Ff33 on Dec 02:


This bug is about Flash crashes in FF36, not FF33, file a new bug, please.

Comment 79

[Suburp]

didn't see this one for a while until the latest stable 36.0 came out. Took only a few hours to show up:
https://crash-stats.mozilla.com/report/index/b840c375-976d-40c9-8170-0ee682150227
(https://bugzilla.mozilla.org/show_bug.cgi?id=1137664)

Comment 80

[Suburp]

(In reply to Loic from comment #78)
> (In reply to Suburp from comment #76)
> > Are you sure this has been solved? seems to do the same in V36, too.
> > 
> > https://crash-stats.mozilla.com/report/index/4a221aa4-146b-4558-a467-
> > 9766a2141202
> > 
> > Still occurs in Ff33 on Dec 02:
> 
> 
> This bug is about Flash crashes in FF36, not FF33, file a new bug, please.

Loic, filed those separately, but this is a 36.0 issue:
https://crash-stats.mozilla.com/report/index/b840c375-976d-40c9-8170-0ee682150227

Comment 81

[Loic]

AFAIK, this Flash crash signature is generic, not sure if it would help a lot. Look at the bug report list, it's huge.