Closed Bug 561019 Opened 15 years ago Closed 12 years ago

Plugin Flash waveOutOpen hang in F328480546

Categories

(Core Graveyard :: Plug-ins, defect, P3)

Product:
Core Graveyard
Component:
Plug-ins
Platform:
x86
Windows 7
Type:
defect

Tracking

(blocking1.9.2 -, status1.9.2 wanted, status1.9.1 unaffected)

Status:
RESOLVED WORKSFORME
Tracking Flags:
Tracking Status
blocking1.9.2 --- -
status1.9.2 --- wanted
status1.9.1 --- unaffected

People

(Reporter: jimm, Unassigned)

References

(
URL
)

Details

(Keywords: hang)

Crash Data

Attachments

(2 files)

test plugin patch
5.67 KB, patch
Details | Diff | Splinter Review
about:config data per requested
3.13 KB, text/plain
Details 
We have quite a few of these with plugins, likely related to playing music or web cams. The waveOutOpen call can send status messages to a window or thread. Those can then trigger other events, such as painting. It would be nice if we could get some uri for some of these crashes. 

http://msdn.microsoft.com/en-us/library/dd743866(VS.85).aspx

http://crash-stats.mozilla.com/report/index/ec3f5a82-22bb-488e-bd4c-150cf2100418

0  	ntdll.dll  	ntdll.dll@0x1f871  	
1 	kernel32.dll 	kernel32.dll@0x11183 	
2 	kernel32.dll 	kernel32.dll@0x11137 	
3 	wdmaud.drv 	wdmaud.drv@0x4d61 	
4 	wdmaud.drv 	wdmaud.drv@0x52e8 	
5 	wdmaud.drv 	wdmaud.drv@0xed39 	
6 	wdmaud.drv 	wdmaud.drv@0xe29f 	
7 	wdmaud.drv 	wdmaud.drv@0x962c 	
8 	wdmaud.drv 	wdmaud.drv@0x4b1c 	
9 	winmm.dll 	waveOutOpen 	
10 	msacm32.drv 	msacm32.drv@0x1bde 	
11 	msacm32.drv 	msacm32.drv@0x1fe6 	
12 	msacm32.drv 	msacm32.drv@0x1460 	
13 	winmm.dll 	waveOutOpen 	
14 	NPSWF32.dll 	NPSWF32.dll@0x198617 	
15 	NPSWF32.dll 	NPSWF32.dll@0xca253 	
16 	NPSWF32.dll 	NPSWF32.dll@0x131806 	
17 	NPSWF32.dll 	NPSWF32.dll@0x12548e 	
18 	NPSWF32.dll 	NPSWF32.dll@0x111693 	
19 	NPSWF32.dll 	NPSWF32.dll@0x1825e5 	
20 	NPSWF32.dll 	NPSWF32.dll@0x113627
Jim, I missed this one along the way. I've seen a fair number of waveOutOpen hangs, but I thought that the audio device was taking too long to respond: I didn't realize that it was sending messages. Let me try to get you some more data from the hang stats.
blocking1.9.2: --- → ?
Component: IPC → Plug-ins
QA Contact: ipc → plugins
Please renominate if this turns out to be our problem.
blocking1.9.2: ? → -
The signature isn't going to have waveOutOpen, the actual hang ends up in ZwWaitForSingleObject, I think. The one example gleaned from my hang stats is http://crash-stats.mozilla.com/report/index/a3693123-f09a-45b2-bd8b-73f492100519
status1.9.1: --- → unaffected
status1.9.2: --- → wanted
Summary: [OOPP] waveOutOpen hangs → Plugin waveOutOpen hangs [@ hang | WaitForSingleObjectEx | WaitForSingleObjectExImplementation | WaitForSingleObject | CJob::Wait(unsigned long) ]
Summary: Plugin waveOutOpen hangs [@ hang | WaitForSingleObjectEx | WaitForSingleObjectExImplementation | WaitForSingleObject | CJob::Wait(unsigned long) ] → Plugin waveOutOpen hangs [@ hang | WaitForSingleObjectEx | WaitForSingleObjectExImplementation | WaitForSingleObject | CJob::Wait(unsigned long) ] | [@ hang | WaitForSingleObjectEx | WaitForSingleObject | CJob::Wait(unsigned long) ]
Crash Signature: [@ hang | WaitForSingleObjectEx | WaitForSingleObjectExImplementation | WaitForSingleObject | CJob::Wait(unsigned long) ] [@ hang | WaitForSingleObjectEx | WaitForSingleObject | CJob::Wait(unsigned long) ]
Crash Signature: [@ hang | WaitForSingleObjectEx | WaitForSingleObjectExImplementation | WaitForSingleObject | CJob::Wait(unsigned long) ] [@ hang | WaitForSingleObjectEx | WaitForSingleObject | CJob::Wait(unsigned long) ] → [@ hang | WaitForSingleObjectEx | WaitForSingleObjectExImplementation | WaitForSingleObject | CJob::Wait(unsigned long) ] [@ hang | WaitForSingleObjectEx | WaitForSingleObject | CJob::Wait(unsigned long) ] [@ hang | kernelbase.dll@0x103a0 ]
Crash Signature: [@ hang | WaitForSingleObjectEx | WaitForSingleObjectExImplementation | WaitForSingleObject | CJob::Wait(unsigned long) ] [@ hang | WaitForSingleObjectEx | WaitForSingleObject | CJob::Wait(unsigned long) ] [@ hang | kernelbase.dll@0x103a0 ] → kernelbase.dll@0x179b ] [@ hang | kernelbase.dll@0x103a0 ] [@ hang | WaitForSingleObjectEx | WaitForSingleObjectExImplementation | WaitForSingleObject | CJob::Wait(unsigned long) ] [@ hang | WaitForSingleObjectEx | WaitForSingleObject | CJob::Wait(unsig…
Assignee: nobody → jmathies
Crash Signature: kernelbase.dll@0x179b ] [@ hang | kernelbase.dll@0x103a0 ] → kernelbase.dll@0x179b ]
Crash Signature: kernelbase.dll@0x179b ] → kernelbase.dll@0x179b ] [@ hang | CreateEventExW ]
Summary: Plugin waveOutOpen hangs [@ hang | WaitForSingleObjectEx | WaitForSingleObjectExImplementation | WaitForSingleObject | CJob::Wait(unsigned long) ] | [@ hang | WaitForSingleObjectEx | WaitForSingleObject | CJob::Wait(unsigned long) ] → Plugin waveOutOpen hangs
Posting some notes on this. I've been experimenting with our test plugin to try to reproduce. I have caught at least one hang occasionally, an exact cause is hard to decipher. It appears windows is waiting for Windowing events to get processed in the parent, but it doesn't appear as if the events being processed are specific to the call the child is making.

General data:

- both windowed and windowless

- always starts with a parent side call into the child, which may or may not get answered before the hang.

- the child is attempting to open an audio device. Windows waits for that device to respond / be ready / whatever which results in a call to WaitForSingleObject.

- The parent at this point is always in WaitForNotify and is spinning (the timeout is picked up).

parent entry points:

mozilla::plugins::PPluginScriptableObjectParent::CallHasProperty

mozilla::plugins::PPluginScriptableObjectParent::CallInvoke

* mozilla::plugins::PPluginInstanceParent::CallUpdateWindow

* mozilla::plugins::PPluginInstanceParent::CallPBrowserStreamConstructo

mozilla::plugins::PluginInstanceParent::NPP_HandleEvent

child entry points:

mozilla::plugins::PluginScriptableObjectChild::AnswerInvoke

* mozilla::plugins::PluginInstanceChild::FlashThrottleAsyncMsg::Run

[
  all child hangs end with:
  F328480546________________________________________________________
  waveOutOpen
  WaitForSingleObject
  ..
  hang
]


The * lines are the most common. I don't think the hang is specific to any of these calls though, these are just very common callouts from the parent to the child.
I'm going to take a break from this and move on to some other things. I've been working with this patch and a test case that calls it on a timer repeatedly. I've tested on single and multi-core systems, release and debug builds, with various other factors thrown in such as youtube video. Overall I haven't been able to reproduce anything reliably. 

It might be possible to reproduce this if the test plugin was involved in some of the chatter I've tried to create with other plugins - namely streaming a lot of data on top of the calls to openWaveAudio. I might try to reproduce something like that in the future. Generally though my original guess that this was caused by some sort of a message hang does not appear to be the case.
I forgot to mention, another test I've tried:  a timer in the plugin which made this call repeatedly with independent calls from the parent to other unrelated plugins calls. This simulates the case where the flash throttle message event makes the waveOutOpen call. I didn't have any luck there. 

I think our best hope here is to wait and see if someone posts a url or test case somewhere that reproduces these stacks.
With combined signatures, it is #7 top crasher in 7.0.1 and #8 in 8.0b1.
reference: 2497eaf9-26b0-46ed-ae9e-14af92111011
I noticed when this happened another crash report was generated at the same time if it helps ( Bug 626900 - Firefox 4.0b9 Crash Report [@ hang | mozilla::plugins::PPluginInstanceParent::CallNPP_Destroy(short*) ]  ) 

I had multiple tabs open when this occurred so i cannot say what the site was. I did notice this happen with a live streaming flash site; cant remember what it was since i only went there once. Sorry.
Carm, you will get 2 reports when this happens, a browser side and a plugin side report so they are really a pair.
Sheila, Do you know what is going on with the flash player constantly crashing the plugin-container.exe? It is very frustrating being a long time Firefox user, and this keeps happening. I have tested 4.01 with the latest Flash, and there are no issues. On Versions6, 7, 8b, 9a I constantly experience the plugin-container.exe crashing.

Can you shed some light on this, is it going to be corrected, or maybe version 4 of the plugin container should be implemented of future versions, etc...

All i know is that after 5 Firefox suffers constant crashing.

Thank You in Advance
(In reply to Carm from comment #9)
> reference: 2497eaf9-26b0-46ed-ae9e-14af92111011

Carm, any chance you might be able to supply a web site / set of steps to reproduce for this type of crash?
Ok, they are live streaming sites using flash players. I have two sites open for several hours, overnight actually one is: http://www.camzone.com/baycam which i just browsed through the cams overnight and let run, and the other is http://www.stickam.com, which i let stream too. Both of these were in two different tabs in the same window. As far as what site crashed it, i honestly do not know. Both streams crashed at the same time. I was looking through the camzone one just this morning, then i clicked on the stickam tab, and when to go click on a stream is when it actually crashed. Honestly the crashes apparently seem to happen randomly. It seems to happen when initializing a stream, but not right away when you are on a site, so you have to be viewing for a while, and then browsing streams as well, but as i said it is totally random, and really hard for me to try to figure out. There are two crash reports generated this morning when this happened:


https://crash-stats.mozilla.com/report/index/bp-22704528-1aed-44a4-b0c3-8f1e22111029

https://crash-stats.mozilla.com/report/index/bp-ef1110cb-3c27-4e48-89fa-8e9e72111029

These crash reports tends to vary as far as what they are, for example these two crashes were not the ones in this bug, but other times could be. I really hope this helps out
I have some more information. I got a freeze recently and an actual crash on the http://www.stickam.com site when browsing streams. The following two crash reports caused a actual plugin container crash:

https://crash-stats.mozilla.com/report/index/bp-837a359d-6f23-43f7-823f-36c202111030

https://crash-stats.mozilla.com/report/index/bp-a06a83db-e1e7-416b-a68a-766b42111030

The next two reports didn't cause a crash, the browser froze for about 30 seconds or whatever the timeout is, and did generate two crash reports that were not being asked to submit, but rather i had checked about:crashes manually to submit:

https://crash-stats.mozilla.com/report/index/bp-12d762ee-ff50-4b15-a7e7-9e8602111030

https://crash-stats.mozilla.com/report/index/bp-b167d862-b339-498d-a713-ec30b2111030

once again i hope this helps put an end to this craziness.

Please advise
Carm, I am going to have someone look into this more thoroughly.
(In reply to Carm from comment #15)
> I have some more information. I got a freeze recently and an actual crash on
> the http://www.stickam.com site when browsing streams. The following two
> crash reports caused a actual plugin container crash:
> 
> https://crash-stats.mozilla.com/report/index/bp-837a359d-6f23-43f7-823f-
> 36c202111030
> 
> https://crash-stats.mozilla.com/report/index/bp-a06a83db-e1e7-416b-a68a-
> 766b42111030

This matches this bug - parent side handle event (mouse move) that on the child side calls waveOutOpen triggering a plugin hang. Looks like the cause is a message back to the parent window procedure.

> The next two reports didn't cause a crash, the browser froze for about 30
> seconds or whatever the timeout is, and did generate two crash reports that
> were not being asked to submit, but rather i had checked about:crashes
> manually to submit:
> 
> https://crash-stats.mozilla.com/report/index/bp-12d762ee-ff50-4b15-a7e7-
> 9e8602111030
> 
> https://crash-stats.mozilla.com/report/index/bp-b167d862-b339-498d-a713-
> ec30b2111030

bug 626900, on the parent's call to NPP_Destroy, the child enters a WaitForSingleObject call via flash and hangs, likely another messaging deadlock. It's hard to predict / reproduce these since we can't predict what Windows does in these calls.

Knowing though that http://www.stickam.com is a common culprit may help. Maybe we can get some qa folks poke around on this site?

Carm - do you know if these happen when you are viewing the live cam streams, replay of cam streams, or both?
It never happens once a stream is acquired ( i am in a session ), mostly when i go to click on a stream, and a few sometimes when i back out of one to the main menu of streams. 

I also use minus.com which is a storage site, and people throw video up there ( plays with flash ). I know its not a live stream, however, I did experience freezing on site. It didn't produce this exact bug this time. If it does i will post it. 

Can we change the dom.ipc.plugins.processLaunchTimeoutSecs to say 15 seconds. 45 seconds is way too long to wait for a crash/hang, and is more tolerable.

Thanks
Entry and exit hangs make perfect sense based on the crashes you posted - one is involved in opening an audio session, the other is involved in destroying a flash instance.
Great, i hope this helps resolve these adobe flash related crashes and hangs :)
Wanted to pass on some more info. Had a script stop responding and a prompt cam up and asked to stop script or continue on http://www.pandora.com . This happened when i un-paused the music after about 45 minutes on pause. After i shoosen stopt script it stated that adobe flash has crashed. Here are the reports:

https://crash-stats.mozilla.com/report/index/bp-e525ae91-d661-4f48-a301-d744f2111102

https://crash-stats.mozilla.com/report/index/bp-d771c7e8-a13f-498a-b2a8-d4ffb2111102

I do not know if you want to keep reporting this issue with different sites, or do you have enough to go on? 

Thanks
Carm, if you find additional sites like stickam and pandora where this is a common problem, please post those. That helps us in looking for good test cases.

Also, would you mind posting your about:support data as a text file attachment to this bug?
i did make some changes to the timeouts so they would not hang so long during an actual crash
Any progress on this issue, anyone?
Is this flash issue getting worked on?
(In reply to Carm from comment #29)
> Is this flash issue getting worked on?

Apparently not. I've been getting this crash with Flash about 10 times a day on Firefox, on two different computers, for several months (religiously updating both Firefox and Flash whenever asked). It's no longer a usable experience for me, so today I'm reluctantly moving to Chrome. Hopefully one day this will be fixed.
Severity: normal → critical
Keywords: hang
We would love to find a fix, but unfortunately we haven't found a test case that reliably reproduces the problem, and we can't reproduce it in-house with various tests. If anyone knows of a way to reproduce this reliably, please post the details.
Is it really the case that there's no other way to diagnose this other than a reproducible test case? Can we get a half dozen of our best and brightest to do some code inspection and make some guesses as to what might be causing this? 

We can't keep sitting on high-volume issues like this hoping to get a reproducible testcase. It may never come, meanwhile our users are suffering.
Josh, Jim. Is this the sort of hang issue that work from bug 651192 is going to address?
(In reply to Sheila Mooney from comment #34)
> Josh, Jim. Is this the sort of hang issue that work from bug 651192 is going
> to address?

Doubtful since this is not related to painting it's related to sound playback in the flash plugin. While opening the audio device which can block a bit, Windows does background processing including event processing. The most likely issue here is event delivery to the chrome process while the plugin is blocked on the waveOutOpen call which is usually the result of some user action or script execution. Comment 5 has a list of the most common chrome/child entry points.
(In reply to Asa Dotzler [:asa] from comment #33)
> Is it really the case that there's no other way to diagnose this other than
> a reproducible test case? Can we get a half dozen of our best and brightest
> to do some code inspection and make some guesses as to what might be causing
> this? 
> 
> We can't keep sitting on high-volume issues like this hoping to get a
> reproducible testcase. It may never come, meanwhile our users are suffering.

I would agree with Asa, this needs to be moved on in some way, and allowing the end user to suffer or find another browser is not really acceptable. Since the release of v8, I have 35 crash reports. 14 of them are this particular crash.

As i have said before. FF4.01 didn't have this issue. Maybe instead of fumbling around, revert to the 4.01 plugin container as it is unaffected by new versions of flash. Something clearly changed from v4.01 plugin container. We seem to be forgetting that, and as Asa stated:

" We can't keep sitting on high-volume issues like this hoping to get a
reproducible testcase. It may never come, meanwhile our users are suffering. "

Something needs to be done as I am already using reluctantly Chrome more frequently since FF cannot seem to handle plugins correctly anymore.

I hope a decision is made about how to rectify this soon
Carm - thanks for your patience on this and for engaging with us to troubleshoot the issue. Let me take another stab getting something moving on this.

Jim - since we have found someone who runs into this frequently, although not reliably, is there any kind debug build, diagnostic tool, anything he could run that might give us some better information?
If it helps, I run into this on both my laptop and my desktop at home (both Realtek audio of some kind) as well as over remote desktop to a Win 7 x64 install running on ESXi 5. If the thought is that the audio driver has some effect on this bug, you might try connecting via Remote Desktop with the audio set to "Play on this computer". That should (I believe) take the installed sound card completely out of the picture and leave you with a system that will randomly experience this bug.
Keywords: topcrash
This signature paired with the browser signature hang | mozilla::plugins::PPluginInstanceParent::CallNPP_HandleEvent(mozilla::plugins::NPRemoteEvent const&, short*) is the #5 hang pair on FF 9.0. Several bugs with the later signature bug 688992, bug 647400, bug 540094. 

https://crash-analysis.mozilla.com/rkaiser/2011-12-08/2011-12-08.firefox.9.0.flashhangdetails.html
https://crash-analysis.mozilla.com/rkaiser/2011-12-12/2011-12-12.firefox.9.0.flashhangdetails.html
RE:

"Scoobidiver 2011-12-21 05:49:42 EST 
Duplicate of this bug: 711922"

are we saying that this bug; 561019 is resolved?
scratch that....Guess not, present in FF 9.01.
It's a Flash hang:
Frame 	Module 	Signature [Expand] 	Source
0 	ntdll.dll 	NtWaitForSingleObject 	
1 	ntdll.dll 	NtWaitForSingleObject 	
2 	KERNELBASE.dll 	CreateEventExW 	
3 	kernel32.dll 	WaitForSingleObjectExImplementation 	
4 	kernel32.dll 	WaitForSingleObject 	
5 	wdmaud.drv 	CJob::Wait 	
6 	wdmaud.drv 	CWorker::SubmitAndWait 	
7 	wdmaud.drv 	CWaveHandle::Open 	
8 	wdmaud.drv 	CWxEndpoint::OpenHandle 	
9 	wdmaud.drv 	CWxd::Open 	
10 	wdmaud.drv 	wodMessage 	
11 	winmm.dll 	waveOutOpen 	
12 	msacm32.drv 	mapDriverOpenWave 	
13 	msacm32.drv 	mapWaveOpen 	
14 	msacm32.drv 	wodMessage 	
15 	winmm.dll 	waveOutOpen 	
16 	NPSWF32.dll 	F328480546________________________________________________________ 	F357347316_________________________________________________:212
17 	NPSWF32.dll 	F1718241201_____________________ 	F_569723929____________________________________:839
18 	NPSWF32.dll 	F_964402326_______________________________________ 	F_569723929____________________________________:2151
19 	NPSWF32.dll 	F1401221563___________________________________________ 	F_502805826________________________________________:15141
20 	NPSWF32.dll 	F450165255___________________________ 	F_502805826________________________________________:5449
21 	NPSWF32.dll 	F1295919189______________________________________ 	F723216689_________________________________________:1823
22 	NPSWF32.dll 	F_1311131038_________________________________________ 	F723216689_________________________________________:3548
23 	NPSWF32.dll 	F_2025036863______________________ 	F_1055812748____________________________________:12049
24 	NPSWF32.dll 	F_1995396427_________________ 	F_1055812748____________________________________:12879
25 	NPSWF32.dll 	F1546913566_____________________________________ 	F806924378_____________________________________________________________________:3928
26 	NPSWF32.dll 	F957328252________________________ 	F806924378_____________________________________________________________________:620
27 	user32.dll 	InternalCallWinProc 	
28 	user32.dll 	UserCallWinProcCheckWow 	
29 	user32.dll 	CallWindowProcAorW 	
30 	user32.dll 	CallWindowProcW 	
31 	xul.dll 	mozilla::plugins::PluginInstanceChild::FlashThrottleAsyncMsg::Run 	dom/plugins/ipc/PluginInstanceChild.cpp:2024
...
Summary: Plugin waveOutOpen hangs → Plugin Flash waveOutOpen hang in F328480546
FAIL! #3 crasher

Imagine that. Mozilla needs to abandon this plugin container period, it is nothing but problems! WAKE UP MOZILLA

https://crash-stats.mozilla.com/report/index/bp-6157b4ad-f8cd-4840-ac1f-998b92120211
Crash Signature: kernelbase.dll@0x179b ] [@ hang | CreateEventExW ] → kernelbase.dll@0x179b ] [@ hang | CreateEventExW ] [@ hang | NtUserMessageCall | NtUserMessageCall | FindAccResource]
I also get this crash regularly. If there is anything I can do to help, let me know. I can build my own copies of Firefox if there is anything you want me to try, or if there is code you want me to try stepping through. I'm also going to attempt to run Chrome more often to see if Flash still crashes over there
can conform this also happens on windows
sorry vista
I'm not seeing any of the existing signatures attached to this bug in Flash 11.3 yet, but this may have shifted to different signatures, we need to keep watching out for it.
Adobe may have fixed this or morphed the signature with Flash 11.3. I'll take a task to check crash-stats in a bit.
Assignee: jmathies → benjamin
Keywords: topcrash
I don't see these showing up any more.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → WORKSFORME
There are still crashes in 19.0.2 (#325 crasher) and 20.0 Beta at a low volume: https://crash-stats.mozilla.com/report/list?signature=hang+|+WaitForSingleObjectEx+|+WaitForSingleObject+|+CJob%3A%3AWait%28unsigned+long%29
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
Assignee: benjamin → nobody
Priority: -- → P3
(In reply to Scoobidiver from comment #52)
> There are still crashes in 19.0.2 (#325 crasher) and 20.0 Beta at a low
> volume

With what Flash version(s)? If none with 11.6 or 11.7, then I think we can safely let go of this bug.
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #53)
> With what Flash version(s)? If none with 11.6 or 11.7, then I think we can
> safely let go of this bug.
There are only 5 crashes with Flash 11.6.602.180.
Status: REOPENED → RESOLVED
Closed: 12 years ago → 12 years ago
Resolution: --- → WORKSFORME
I'm running a Dell XPS 14, Linux Mint, Tor Browser bundle from the Ubuntu PPA (bundled firefox 17) and I recieve a complete crash everytime I load myfreecams.com.

The problem might be fixed with flash 11.7, but linux seems to stop at 11.2...

So.. not resolved?
obnoxious, this Flash crash has never happened on Linux.
Product: Core → Core Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: