Closed Bug 561019 Opened 14 years ago Closed 11 years ago
Plugin Flash wave
Out Open hang in F328480546
We have quite a few of these with plugins, likely related to playing music or web cams. The waveOutOpen call can send status messages to a window or thread. Those can then trigger other events, such as painting. It would be nice if we could get some uri for some of these crashes. http://msdn.microsoft.com/en-us/library/dd743866(VS.85).aspx http://crash-stats.mozilla.com/report/index/ec3f5a82-22bb-488e-bd4c-150cf2100418 0 ntdll.dll ntdll.dll@0x1f871 1 kernel32.dll kernel32.dll@0x11183 2 kernel32.dll kernel32.dll@0x11137 3 wdmaud.drv wdmaud.drv@0x4d61 4 wdmaud.drv wdmaud.drv@0x52e8 5 wdmaud.drv wdmaud.drv@0xed39 6 wdmaud.drv wdmaud.drv@0xe29f 7 wdmaud.drv wdmaud.drv@0x962c 8 wdmaud.drv wdmaud.drv@0x4b1c 9 winmm.dll waveOutOpen 10 msacm32.drv msacm32.drv@0x1bde 11 msacm32.drv msacm32.drv@0x1fe6 12 msacm32.drv msacm32.drv@0x1460 13 winmm.dll waveOutOpen 14 NPSWF32.dll NPSWF32.dll@0x198617 15 NPSWF32.dll NPSWF32.dll@0xca253 16 NPSWF32.dll NPSWF32.dll@0x131806 17 NPSWF32.dll NPSWF32.dll@0x12548e 18 NPSWF32.dll NPSWF32.dll@0x111693 19 NPSWF32.dll NPSWF32.dll@0x1825e5 20 NPSWF32.dll NPSWF32.dll@0x113627
Jim, I missed this one along the way. I've seen a fair number of waveOutOpen hangs, but I thought that the audio device was taking too long to respond: I didn't realize that it was sending messages. Let me try to get you some more data from the hang stats.
blocking1.9.2: --- → ?
Component: IPC → Plug-ins
QA Contact: ipc → plugins
Please renominate if this turns out to be our problem.
blocking1.9.2: ? → -
I wonder if crash stats is failing me here, I can't find any of these stacks: http://crash-stats.mozilla.com/query/query?product=Firefox&version=Firefox%3A3.6.4&platform=windows&range_value=2&range_unit=weeks&date=05%2F24%2F2010+07%3A43%3A40&query_search=signature&query_type=contains&query=waveOutOpen&build_id=&process_type=any&hang_type=any&do_query=1
The signature isn't going to have waveOutOpen, the actual hang ends up in ZwWaitForSingleObject, I think. The one example gleaned from my hang stats is http://crash-stats.mozilla.com/report/index/a3693123-f09a-45b2-bd8b-73f492100519
Summary: [OOPP] waveOutOpen hangs → Plugin waveOutOpen hangs [@ hang | WaitForSingleObjectEx | WaitForSingleObjectExImplementation | WaitForSingleObject | CJob::Wait(unsigned long) ]
Summary: Plugin waveOutOpen hangs [@ hang | WaitForSingleObjectEx | WaitForSingleObjectExImplementation | WaitForSingleObject | CJob::Wait(unsigned long) ] → Plugin waveOutOpen hangs [@ hang | WaitForSingleObjectEx | WaitForSingleObjectExImplementation | WaitForSingleObject | CJob::Wait(unsigned long) ] | [@ hang | WaitForSingleObjectEx | WaitForSingleObject | CJob::Wait(unsigned long) ]
Crash Signature: [@ hang | WaitForSingleObjectEx | WaitForSingleObjectExImplementation | WaitForSingleObject | CJob::Wait(unsigned long) ] [@ hang | WaitForSingleObjectEx | WaitForSingleObject | CJob::Wait(unsigned long) ]
Crash Signature: [@ hang | WaitForSingleObjectEx | WaitForSingleObjectExImplementation | WaitForSingleObject | CJob::Wait(unsigned long) ] [@ hang | WaitForSingleObjectEx | WaitForSingleObject | CJob::Wait(unsigned long) ] → [@ hang | WaitForSingleObjectEx | WaitForSingleObjectExImplementation | WaitForSingleObject | CJob::Wait(unsigned long) ] [@ hang | WaitForSingleObjectEx | WaitForSingleObject | CJob::Wait(unsigned long) ] [@ hang | kernelbase.dll@0x103a0 ]
Crash Signature: [@ hang | WaitForSingleObjectEx | WaitForSingleObjectExImplementation | WaitForSingleObject | CJob::Wait(unsigned long) ] [@ hang | WaitForSingleObjectEx | WaitForSingleObject | CJob::Wait(unsigned long) ] [@ hang | kernelbase.dll@0x103a0 ] → kernelbase.dll@0x179b ] [@ hang | kernelbase.dll@0x103a0 ] [@ hang | WaitForSingleObjectEx | WaitForSingleObjectExImplementation | WaitForSingleObject | CJob::Wait(unsigned long) ] [@ hang | WaitForSingleObjectEx | WaitForSingleObject | CJob::Wait(unsig…
Crash Signature: kernelbase.dll@0x179b ] [@ hang | kernelbase.dll@0x103a0 ] → kernelbase.dll@0x179b ]
Crash Signature: kernelbase.dll@0x179b ] → kernelbase.dll@0x179b ] [@ hang | CreateEventExW ]
Summary: Plugin waveOutOpen hangs [@ hang | WaitForSingleObjectEx | WaitForSingleObjectExImplementation | WaitForSingleObject | CJob::Wait(unsigned long) ] | [@ hang | WaitForSingleObjectEx | WaitForSingleObject | CJob::Wait(unsigned long) ] → Plugin waveOutOpen hangs
Posting some notes on this. I've been experimenting with our test plugin to try to reproduce. I have caught at least one hang occasionally, an exact cause is hard to decipher. It appears windows is waiting for Windowing events to get processed in the parent, but it doesn't appear as if the events being processed are specific to the call the child is making. General data: - both windowed and windowless - always starts with a parent side call into the child, which may or may not get answered before the hang. - the child is attempting to open an audio device. Windows waits for that device to respond / be ready / whatever which results in a call to WaitForSingleObject. - The parent at this point is always in WaitForNotify and is spinning (the timeout is picked up). parent entry points: mozilla::plugins::PPluginScriptableObjectParent::CallHasProperty mozilla::plugins::PPluginScriptableObjectParent::CallInvoke * mozilla::plugins::PPluginInstanceParent::CallUpdateWindow * mozilla::plugins::PPluginInstanceParent::CallPBrowserStreamConstructo mozilla::plugins::PluginInstanceParent::NPP_HandleEvent child entry points: mozilla::plugins::PluginScriptableObjectChild::AnswerInvoke * mozilla::plugins::PluginInstanceChild::FlashThrottleAsyncMsg::Run [ all child hangs end with: F328480546________________________________________________________ waveOutOpen WaitForSingleObject .. hang ] The * lines are the most common. I don't think the hang is specific to any of these calls though, these are just very common callouts from the parent to the child.
I'm going to take a break from this and move on to some other things. I've been working with this patch and a test case that calls it on a timer repeatedly. I've tested on single and multi-core systems, release and debug builds, with various other factors thrown in such as youtube video. Overall I haven't been able to reproduce anything reliably. It might be possible to reproduce this if the test plugin was involved in some of the chatter I've tried to create with other plugins - namely streaming a lot of data on top of the calls to openWaveAudio. I might try to reproduce something like that in the future. Generally though my original guess that this was caused by some sort of a message hang does not appear to be the case.
I forgot to mention, another test I've tried: a timer in the plugin which made this call repeatedly with independent calls from the parent to other unrelated plugins calls. This simulates the case where the flash throttle message event makes the waveOutOpen call. I didn't have any luck there. I think our best hope here is to wait and see if someone posts a url or test case somewhere that reproduces these stacks.
With combined signatures, it is #7 top crasher in 7.0.1 and #8 in 8.0b1.
I noticed when this happened another crash report was generated at the same time if it helps ( Bug 626900 - Firefox 4.0b9 Crash Report [@ hang | mozilla::plugins::PPluginInstanceParent::CallNPP_Destroy(short*) ] ) I had multiple tabs open when this occurred so i cannot say what the site was. I did notice this happen with a live streaming flash site; cant remember what it was since i only went there once. Sorry.
Carm, you will get 2 reports when this happens, a browser side and a plugin side report so they are really a pair.
Sheila, Do you know what is going on with the flash player constantly crashing the plugin-container.exe? It is very frustrating being a long time Firefox user, and this keeps happening. I have tested 4.01 with the latest Flash, and there are no issues. On Versions6, 7, 8b, 9a I constantly experience the plugin-container.exe crashing. Can you shed some light on this, is it going to be corrected, or maybe version 4 of the plugin container should be implemented of future versions, etc... All i know is that after 5 Firefox suffers constant crashing. Thank You in Advance
(In reply to Carm from comment #9) > reference: 2497eaf9-26b0-46ed-ae9e-14af92111011 Carm, any chance you might be able to supply a web site / set of steps to reproduce for this type of crash?
Ok, they are live streaming sites using flash players. I have two sites open for several hours, overnight actually one is: http://www.camzone.com/baycam which i just browsed through the cams overnight and let run, and the other is http://www.stickam.com, which i let stream too. Both of these were in two different tabs in the same window. As far as what site crashed it, i honestly do not know. Both streams crashed at the same time. I was looking through the camzone one just this morning, then i clicked on the stickam tab, and when to go click on a stream is when it actually crashed. Honestly the crashes apparently seem to happen randomly. It seems to happen when initializing a stream, but not right away when you are on a site, so you have to be viewing for a while, and then browsing streams as well, but as i said it is totally random, and really hard for me to try to figure out. There are two crash reports generated this morning when this happened: https://crash-stats.mozilla.com/report/index/bp-22704528-1aed-44a4-b0c3-8f1e22111029 https://crash-stats.mozilla.com/report/index/bp-ef1110cb-3c27-4e48-89fa-8e9e72111029 These crash reports tends to vary as far as what they are, for example these two crashes were not the ones in this bug, but other times could be. I really hope this helps out
I have some more information. I got a freeze recently and an actual crash on the http://www.stickam.com site when browsing streams. The following two crash reports caused a actual plugin container crash: https://crash-stats.mozilla.com/report/index/bp-837a359d-6f23-43f7-823f-36c202111030 https://crash-stats.mozilla.com/report/index/bp-a06a83db-e1e7-416b-a68a-766b42111030 The next two reports didn't cause a crash, the browser froze for about 30 seconds or whatever the timeout is, and did generate two crash reports that were not being asked to submit, but rather i had checked about:crashes manually to submit: https://crash-stats.mozilla.com/report/index/bp-12d762ee-ff50-4b15-a7e7-9e8602111030 https://crash-stats.mozilla.com/report/index/bp-b167d862-b339-498d-a713-ec30b2111030 once again i hope this helps put an end to this craziness. Please advise
Carm, I am going to have someone look into this more thoroughly.
(In reply to Carm from comment #15) > I have some more information. I got a freeze recently and an actual crash on > the http://www.stickam.com site when browsing streams. The following two > crash reports caused a actual plugin container crash: > > https://crash-stats.mozilla.com/report/index/bp-837a359d-6f23-43f7-823f- > 36c202111030 > > https://crash-stats.mozilla.com/report/index/bp-a06a83db-e1e7-416b-a68a- > 766b42111030 This matches this bug - parent side handle event (mouse move) that on the child side calls waveOutOpen triggering a plugin hang. Looks like the cause is a message back to the parent window procedure. > The next two reports didn't cause a crash, the browser froze for about 30 > seconds or whatever the timeout is, and did generate two crash reports that > were not being asked to submit, but rather i had checked about:crashes > manually to submit: > > https://crash-stats.mozilla.com/report/index/bp-12d762ee-ff50-4b15-a7e7- > 9e8602111030 > > https://crash-stats.mozilla.com/report/index/bp-b167d862-b339-498d-a713- > ec30b2111030 bug 626900, on the parent's call to NPP_Destroy, the child enters a WaitForSingleObject call via flash and hangs, likely another messaging deadlock. It's hard to predict / reproduce these since we can't predict what Windows does in these calls. Knowing though that http://www.stickam.com is a common culprit may help. Maybe we can get some qa folks poke around on this site? Carm - do you know if these happen when you are viewing the live cam streams, replay of cam streams, or both?
It never happens once a stream is acquired ( i am in a session ), mostly when i go to click on a stream, and a few sometimes when i back out of one to the main menu of streams. I also use minus.com which is a storage site, and people throw video up there ( plays with flash ). I know its not a live stream, however, I did experience freezing on site. It didn't produce this exact bug this time. If it does i will post it. Can we change the dom.ipc.plugins.processLaunchTimeoutSecs to say 15 seconds. 45 seconds is way too long to wait for a crash/hang, and is more tolerable. Thanks
Entry and exit hangs make perfect sense based on the crashes you posted - one is involved in opening an audio session, the other is involved in destroying a flash instance.
Great, i hope this helps resolve these adobe flash related crashes and hangs :)
Wanted to pass on some more info. Had a script stop responding and a prompt cam up and asked to stop script or continue on http://www.pandora.com . This happened when i un-paused the music after about 45 minutes on pause. After i shoosen stopt script it stated that adobe flash has crashed. Here are the reports: https://crash-stats.mozilla.com/report/index/bp-e525ae91-d661-4f48-a301-d744f2111102 https://crash-stats.mozilla.com/report/index/bp-d771c7e8-a13f-498a-b2a8-d4ffb2111102 I do not know if you want to keep reporting this issue with different sites, or do you have enough to go on? Thanks
Carm, if you find additional sites like stickam and pandora where this is a common problem, please post those. That helps us in looking for good test cases. Also, would you mind posting your about:support data as a text file attachment to this bug?
i did make some changes to the timeouts so they would not hang so long during an actual crash
more information regarding flash crashes: http://sites.garmin.com/etrex/ when leaving this page https://crash-stats.mozilla.com/report/index/bp-89b800b5-18c4-4e4b-a8fd-2bfbb2111105 https://crash-stats.mozilla.com/report/index/bp-98889f52-e958-4596-aacc-905992111105 leaving this page: http://sites.garmin.com/gpsmap62/ https://crash-stats.mozilla.com/report/index/bp-cb3460dc-58c4-4e0a-92f2-181112111105 https://crash-stats.mozilla.com/report/index/bp-e42ebe8d-610b-45a5-b2d2-8c7422111105
Firefox v8.0 hang when nav away from this page: http://minus.com/mbkgV9iWFX#1 https://crash-stats.mozilla.com/report/index/bp-3df87ef6-28d4-413e-b833-f7ce92111112 https://crash-stats.mozilla.com/report/index/bp-fd629347-55e9-40a1-9c33-105f62111112
Any progress on this issue, anyone?
playing/switching songs on this page, which does contain flash content: http://minus.com/mbcCOyzaYM#1l https://crash-stats.mozilla.com/report/index/bp-7028cd52-47f7-4a03-aa9c-208522111115 https://crash-stats.mozilla.com/report/index/bp-9de4356b-9020-42ae-9e97-104582111115
Is this flash issue getting worked on?
(In reply to Carm from comment #29) > Is this flash issue getting worked on? Apparently not. I've been getting this crash with Flash about 10 times a day on Firefox, on two different computers, for several months (religiously updating both Firefox and Flash whenever asked). It's no longer a usable experience for me, so today I'm reluctantly moving to Chrome. Hopefully one day this will be fixed.
We would love to find a fix, but unfortunately we haven't found a test case that reliably reproduces the problem, and we can't reproduce it in-house with various tests. If anyone knows of a way to reproduce this reliably, please post the details.
Is it really the case that there's no other way to diagnose this other than a reproducible test case? Can we get a half dozen of our best and brightest to do some code inspection and make some guesses as to what might be causing this? We can't keep sitting on high-volume issues like this hoping to get a reproducible testcase. It may never come, meanwhile our users are suffering.
Josh, Jim. Is this the sort of hang issue that work from bug 651192 is going to address?
(In reply to Sheila Mooney from comment #34) > Josh, Jim. Is this the sort of hang issue that work from bug 651192 is going > to address? Doubtful since this is not related to painting it's related to sound playback in the flash plugin. While opening the audio device which can block a bit, Windows does background processing including event processing. The most likely issue here is event delivery to the chrome process while the plugin is blocked on the waveOutOpen call which is usually the result of some user action or script execution. Comment 5 has a list of the most common chrome/child entry points.
(In reply to Asa Dotzler [:asa] from comment #33) > Is it really the case that there's no other way to diagnose this other than > a reproducible test case? Can we get a half dozen of our best and brightest > to do some code inspection and make some guesses as to what might be causing > this? > > We can't keep sitting on high-volume issues like this hoping to get a > reproducible testcase. It may never come, meanwhile our users are suffering. I would agree with Asa, this needs to be moved on in some way, and allowing the end user to suffer or find another browser is not really acceptable. Since the release of v8, I have 35 crash reports. 14 of them are this particular crash. As i have said before. FF4.01 didn't have this issue. Maybe instead of fumbling around, revert to the 4.01 plugin container as it is unaffected by new versions of flash. Something clearly changed from v4.01 plugin container. We seem to be forgetting that, and as Asa stated: " We can't keep sitting on high-volume issues like this hoping to get a reproducible testcase. It may never come, meanwhile our users are suffering. " Something needs to be done as I am already using reluctantly Chrome more frequently since FF cannot seem to handle plugins correctly anymore. I hope a decision is made about how to rectify this soon
Carm - thanks for your patience on this and for engaging with us to troubleshoot the issue. Let me take another stab getting something moving on this. Jim - since we have found someone who runs into this frequently, although not reliably, is there any kind debug build, diagnostic tool, anything he could run that might give us some better information?
If it helps, I run into this on both my laptop and my desktop at home (both Realtek audio of some kind) as well as over remote desktop to a Win 7 x64 install running on ESXi 5. If the thought is that the audio driver has some effect on this bug, you might try connecting via Remote Desktop with the audio set to "Play on this computer". That should (I believe) take the installed sound card completely out of the picture and leave you with a system that will randomly experience this bug.
This signature paired with the browser signature hang | mozilla::plugins::PPluginInstanceParent::CallNPP_HandleEvent(mozilla::plugins::NPRemoteEvent const&, short*) is the #5 hang pair on FF 9.0. Several bugs with the later signature bug 688992, bug 647400, bug 540094. https://crash-analysis.mozilla.com/rkaiser/2011-12-08/2011-12-08.firefox.9.0.flashhangdetails.html https://crash-analysis.mozilla.com/rkaiser/2011-12-12/2011-12-12.firefox.9.0.flashhangdetails.html
RE: "Scoobidiver 2011-12-21 05:49:42 EST Duplicate of this bug: 711922" are we saying that this bug; 561019 is resolved?
scratch that....Guess not, present in FF 9.01.
It's a Flash hang: Frame Module Signature [Expand] Source 0 ntdll.dll NtWaitForSingleObject 1 ntdll.dll NtWaitForSingleObject 2 KERNELBASE.dll CreateEventExW 3 kernel32.dll WaitForSingleObjectExImplementation 4 kernel32.dll WaitForSingleObject 5 wdmaud.drv CJob::Wait 6 wdmaud.drv CWorker::SubmitAndWait 7 wdmaud.drv CWaveHandle::Open 8 wdmaud.drv CWxEndpoint::OpenHandle 9 wdmaud.drv CWxd::Open 10 wdmaud.drv wodMessage 11 winmm.dll waveOutOpen 12 msacm32.drv mapDriverOpenWave 13 msacm32.drv mapWaveOpen 14 msacm32.drv wodMessage 15 winmm.dll waveOutOpen 16 NPSWF32.dll F328480546________________________________________________________ F357347316_________________________________________________:212 17 NPSWF32.dll F1718241201_____________________ F_569723929____________________________________:839 18 NPSWF32.dll F_964402326_______________________________________ F_569723929____________________________________:2151 19 NPSWF32.dll F1401221563___________________________________________ F_502805826________________________________________:15141 20 NPSWF32.dll F450165255___________________________ F_502805826________________________________________:5449 21 NPSWF32.dll F1295919189______________________________________ F723216689_________________________________________:1823 22 NPSWF32.dll F_1311131038_________________________________________ F723216689_________________________________________:3548 23 NPSWF32.dll F_2025036863______________________ F_1055812748____________________________________:12049 24 NPSWF32.dll F_1995396427_________________ F_1055812748____________________________________:12879 25 NPSWF32.dll F1546913566_____________________________________ F806924378_____________________________________________________________________:3928 26 NPSWF32.dll F957328252________________________ F806924378_____________________________________________________________________:620 27 user32.dll InternalCallWinProc 28 user32.dll UserCallWinProcCheckWow 29 user32.dll CallWindowProcAorW 30 user32.dll CallWindowProcW 31 xul.dll mozilla::plugins::PluginInstanceChild::FlashThrottleAsyncMsg::Run dom/plugins/ipc/PluginInstanceChild.cpp:2024 ...
Summary: Plugin waveOutOpen hangs → Plugin Flash waveOutOpen hang in F328480546
FAIL! #3 crasher Imagine that. Mozilla needs to abandon this plugin container period, it is nothing but problems! WAKE UP MOZILLA https://crash-stats.mozilla.com/report/index/bp-6157b4ad-f8cd-4840-ac1f-998b92120211
Crash Signature: kernelbase.dll@0x179b ] [@ hang | CreateEventExW ] → kernelbase.dll@0x179b ] [@ hang | CreateEventExW ] [@ hang | NtUserMessageCall | NtUserMessageCall | FindAccResource]
I also get this crash regularly. If there is anything I can do to help, let me know. I can build my own copies of Firefox if there is anything you want me to try, or if there is code you want me to try stepping through. I'm also going to attempt to run Chrome more often to see if Flash still crashes over there
can conform this also happens on windows
I'm not seeing any of the existing signatures attached to this bug in Flash 11.3 yet, but this may have shifted to different signatures, we need to keep watching out for it.
Adobe may have fixed this or morphed the signature with Flash 11.3. I'll take a task to check crash-stats in a bit.
Assignee: jmathies → benjamin
I don't see these showing up any more.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → WORKSFORME
There are still crashes in 19.0.2 (#325 crasher) and 20.0 Beta at a low volume: https://crash-stats.mozilla.com/report/list?signature=hang+|+WaitForSingleObjectEx+|+WaitForSingleObject+|+CJob%3A%3AWait%28unsigned+long%29
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
(In reply to Scoobidiver from comment #52) > There are still crashes in 19.0.2 (#325 crasher) and 20.0 Beta at a low > volume With what Flash version(s)? If none with 11.6 or 11.7, then I think we can safely let go of this bug.
(In reply to Robert Kaiser (:firstname.lastname@example.org) from comment #53) > With what Flash version(s)? If none with 11.6 or 11.7, then I think we can > safely let go of this bug. There are only 5 crashes with Flash 11.6.602.180.
Status: REOPENED → RESOLVED
Closed: 11 years ago → 11 years ago
Resolution: --- → WORKSFORME
I'm running a Dell XPS 14, Linux Mint, Tor Browser bundle from the Ubuntu PPA (bundled firefox 17) and I recieve a complete crash everytime I load myfreecams.com. The problem might be fixed with flash 11.7, but linux seems to stop at 11.2... So.. not resolved?
obnoxious, this Flash crash has never happened on Linux.
You need to log in before you can comment on or make changes to this bug.