Last Comment Bug 802355 - crash spike in mozilla::plugins::PluginModuleParent::StreamCast with abort message: "Corrupted plugin stream data.: file e:/builds/moz2_slave/m-cen-w32-ntly/build/dom/plugins/ipc/PluginModuleParent.cpp, line 844"
: crash spike in mozilla::plugins::PluginModuleParent::StreamCast with abort me...
Status: RESOLVED FIXED
[qa?]
: crash, csectype-dos, regression, steps-wanted, topcrash
Product: Core
Classification: Components
Component: Plug-ins (show other bugs)
: 17 Branch
: All All
: -- blocker (vote)
: mozilla19
Assigned To: John Schoenick [:johns]
:
Mentors:
Depends on:
Blocks: 556643 787778
  Show dependency treegraph
 
Reported: 2012-10-16 14:28 PDT by Scoobidiver (away)
Modified: 2013-02-07 06:27 PST (History)
12 users (show)
See Also:
Crash Signature:
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---
+
fixed
+
fixed
+
verified


Attachments
Handle failures in OnStartRequest when spawning plugins (1.57 KB, patch)
2012-10-19 12:27 PDT, John Schoenick [:johns]
jaas: review+
lukasblakk+bugs: approval‑mozilla‑aurora+
lukasblakk+bugs: approval‑mozilla‑beta+
john: checkin+
Details | Diff | Splinter Review

Description Scoobidiver (away) 2012-10-16 14:28:46 PDT
It started spiking from 19.0a1/20121015. The regression range for the spike is:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=57304bbf9c0e&tochange=942ed5747b63
It might be a regression from bug 626245 or bug 787778.

Signature 	mozalloc_abort(char const* const) | NS_DebugBreak_P | nsRefPtr<nsHttpHandler>::nsRefPtr<nsHttpHandler>(nsHttpHandler*) | mozilla::plugins::PluginModuleParent::StreamCast(_NPP*, _NPStream*) More Reports Search
UUID	197b855b-ff99-45a7-bf68-e35302121016
Date Processed	2012-10-16 19:51:03
Uptime	7150
Last Crash	6.3 weeks before submission
Install Age	2.0 hours since version was first installed.
Install Time	2012-10-16 17:51:33
Product	Firefox
Version	19.0a1
Build ID	20121015030612
Release Channel	nightly
OS	Windows NT
OS Version	6.0.6001 Service Pack 1
Build Architecture	x86
Build Architecture Info	GenuineIntel family 6 model 23 stepping 10
Crash Reason	EXCEPTION_BREAKPOINT
Crash Address	0x68df1999
App Notes 	
AdapterVendorID: 0x1039, AdapterDeviceID: 0x6351, AdapterSubsysID: 08011558, AdapterDriverVersion: 7.14.10.5170
D3D10 Layers? D3D10 Layers- D3D9 Layers? D3D9 Layers- xpcom_runtime_abort(###!!! ABORT: Corrupted plugin stream data.: file e:/builds/moz2_slave/m-cen-w32-ntly/build/dom/plugins/ipc/PluginModuleParent.cpp, line 844)
EMCheckCompatibility	True
Adapter Vendor ID	0x1039
Adapter Device ID	0x6351
Total Virtual Memory	2147352576
Available Virtual Memory	1679745024
System Memory Use Percentage	43
Available Page File	5151834112
Available Physical Memory	1744703488

Frame 	Module 	Signature 	Source
0 	mozalloc.dll 	mozalloc_abort 	memory/mozalloc/mozalloc_abort.cpp:23
1 	xul.dll 	NS_DebugBreak_P 	xpcom/base/nsDebugImpl.cpp:410
2 	xul.dll 	nsRefPtr<nsHttpHandler>::nsRefPtr<nsHttpHandler> 	obj-firefox/dist/include/nsAutoPtr.h:898
3 	xul.dll 	mozilla::plugins::PluginModuleParent::StreamCast 	dom/plugins/ipc/PluginModuleParent.cpp:844
4 	xul.dll 	mozilla::plugins::PluginModuleParent::NPP_WriteReady 	dom/plugins/ipc/PluginModuleParent.cpp:662
5 	xul.dll 	nsNPAPIPluginStreamListener::OnDataAvailable 	dom/plugins/base/nsNPAPIPluginStreamListener.cpp:562
6 	xul.dll 	nsPluginStreamListenerPeer::OnDataAvailable 	dom/plugins/base/nsPluginStreamListenerPeer.cpp:931
7 	xul.dll 	nsScriptSecurityManager::SubjectPrincipalIsSystem 	caps/src/nsScriptSecurityManager.cpp:1838
8 	xul.dll 	mozilla::net::nsHttpChannel::OnDataAvailable 	netwerk/protocol/http/nsHttpChannel.cpp:5087

More reports at:
https://crash-stats.mozilla.com/query/query?product=Firefox&version=Firefox%3A19.0a1&query_search=signature&query_type=contains&query=mozilla%3A%3Aplugins%3A%3APluginModuleParent%3A%3AStreamCast&do_query=1
Comment 1 Scoobidiver (away) 2012-10-17 01:31:06 PDT
It's #6 top crasher in today's build.
Comment 2 Scoobidiver (away) 2012-10-17 01:48:46 PDT
It's also #4 top crasher in 18.0a2 where it first appeared in 18.0a2/20121015. The Aurora regression range for the spike is:
http://hg.mozilla.org/releases/mozilla-aurora/pushloghtml?fromchange=0f13f4e81d94&tochange=53559d475428
So it's caused either by bug 787778 or bug 801362.
Comment 3 John Schoenick [:johns] 2012-10-17 12:09:04 PDT
A test case would be very helpful here, but I have a few ideas of what might be happening.

Why we are aborting instead of just asserting and closing the channel, however, is beyond me
Comment 4 Benjamin Smedberg [:bsmedberg] 2012-10-17 12:23:24 PDT
How do you know the channel is in a useful consistent state? The abort seems perfectly reasonable to me here.
Comment 5 Benjamin Smedberg [:bsmedberg] 2012-10-17 13:38:38 PDT
Similar to bug 795683, but with a different call trace.
Comment 6 Scoobidiver (away) 2012-10-17 13:54:38 PDT
One comment says:
"so something in mibew.org PHP/JS chat is crashing stock Aurora. used safe mode and it still crashed it when entering into the chat.tpl view. debugger says there's a deprecated js term."

Here are correlations per extension in 18.0a2:
  mozalloc_abort(char const* const) | NS_DebugBreak_P | mozilla::plugins::PluginModuleParent::StreamCast(_NPP*, _NPStream*)|EXCEPTION_BREAKPOINT (168 crashes)
     32% (54/168) vs.  16% (366/2353) {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d} (Adblock Plus, https://addons.mozilla.org/addon/1865)
     16% (27/168) vs.   2% (47/2353) {d40f5e7b-d2cf-4856-b441-cc613eeffbe3} (BetterPrivacy, https://addons.mozilla.org/addon/6623)
     15% (25/168) vs.   2% (58/2353) {635abd67-4fe9-1b23-4f01-e679fa7484c1} (Yahoo! Toolbar, https://addons.mozilla.org/addon/2032)
     14% (24/168) vs.   4% (96/2353) {1018e4d6-728f-4b20-ad56-37578a4de76b} (Flagfox, https://addons.mozilla.org/addon/5791)
     12% (20/168) vs.   3% (67/2353) {DDC359D1-844A-42a7-9AA1-88A850A938A8} (DownThemAll!, https://addons.mozilla.org/addon/201)
     11% (18/168) vs.   2% (42/2353) donottrackplus@abine.com
     96% (161/168) vs.  87% (2053/2353) {972ce4c6-7e08-4474-a285-3208198ce6fd} (Default, https://addons.mozilla.org/addon/8150)
     10% (17/168) vs.   3% (64/2353) firefox@ghostery.com (Ghostery, https://addons.mozilla.org/addon/9609)
     13% (21/168) vs.   5% (121/2353) {D4DD63FA-01E4-46a7-B6B1-EDAB7D6AD389} (Download Statusbar, https://addons.mozilla.org/addon/26)
     10% (17/168) vs.   3% (73/2353) {a0d7ccb3-214d-498b-b4aa-0e8fda9a7bf7} (WOT, https://addons.mozilla.org/addon/3456)
     10% (16/168) vs.   3% (60/2353) {b64982b1-d112-42b5-b1e4-d3867c4533f8}
Comment 7 Scoobidiver (away) 2012-10-19 10:32:01 PDT
It seems it has been uplifted in 17.0 Beta 2 (currently #2 top crasher).
The Beta regression range is:
http://hg.mozilla.org/releases/mozilla-beta/pushloghtml?fromchange=6b83222781e3&tochange=2ba4becf6e35
The only bug that belongs to the three regression ranges is bug 787778.
Comment 8 John Schoenick [:johns] 2012-10-19 12:27:59 PDT
Created attachment 673356 [details] [diff] [review]
Handle failures in OnStartRequest when spawning plugins

I'm still not able to reproduce this, but this is a speculative fix - bug 787778
lost a check for OnStartRequest failures, so we may end up trying to feed
channel data to a stream listener without a plugin.

try:
https://tbpl.mozilla.org/?tree=Try&rev=430efef625a8
Comment 9 Lukas Blakk [:lsblakk] use ?needinfo 2012-10-19 15:17:33 PDT
Will make sure we backout bug 787778 before 17.0b3
Comment 10 Lukas Blakk [:lsblakk] use ?needinfo 2012-10-19 15:17:58 PDT
Unless John can provide a fix for the crash before Monday.
Comment 11 John Schoenick [:johns] 2012-10-20 20:55:16 PDT
Comment on attachment 673356 [details] [diff] [review]
Handle failures in OnStartRequest when spawning plugins

Landed:
https://hg.mozilla.org/mozilla-central/rev/84686086802c

Will keep an eye on if this signature disappears
Comment 12 John Schoenick [:johns] 2012-10-22 12:58:32 PDT
Comment on attachment 673356 [details] [diff] [review]
Handle failures in OnStartRequest when spawning plugins

There appears to be just one crash on the 21st nightly, vs numerous on the 20th

The single crash on the 21st has a different call stack and looks to be unrelated:
https://crash-stats.mozilla.com/report/index/fc835e88-deed-4584-9ced-c525e2121022

I'm fairly sure that is an unrelated stop-plugin issue (bug 767635), unrelated to 787778

[Approval Request Comment]
Regression caused by (bug #): 
bug 787778

User impact if declined: 
#6 Top crasher (non-exploitable)

Testing completed (on m-c, etc.): 
Crashes appear to have stopped after landing on m-c, but no STR for verification

Risk to taking this patch (and alternatives if risky):
Moderate-to-low, handles a otherwise unhandled failure case, but does touch re-entrance sensitive code. Alternative is backing out bug 787778.
Comment 13 Lukas Blakk [:lsblakk] use ?needinfo 2012-10-22 14:41:29 PDT
Comment on attachment 673356 [details] [diff] [review]
Handle failures in OnStartRequest when spawning plugins

Thanks for doing this so fast, we definitely have time to get this into Beta 3 and confirm that the backout option is no longer needed.  Please go ahead with uplift.
Comment 15 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2012-11-06 14:06:27 PST
I see no crash reports for this passed October 16th. Given this is a blocker though, I'm a little concerned that we don't have steps to reproduce. Is there some focused testing QA can do around this in the next Beta to make sure we are okay? Otherwise, I guess we can mark this verified.
Comment 16 John Schoenick [:johns] 2012-11-06 14:30:42 PST
(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #15)
> I see no crash reports for this passed October 16th. Given this is a blocker
> though, I'm a little concerned that we don't have steps to reproduce. Is
> there some focused testing QA can do around this in the next Beta to make
> sure we are okay? Otherwise, I guess we can mark this verified.

Unfortunately I was unable to find a test case here - finding one would likely involve pulling up the URLs involved and finding out what plugin versions were in use -- it seems like the affected sites are using media types (e.g. audio/wav), so there's probably a specific version of a media player (vlc firefox plugin, windows media player, quicktime...) that is causing the crash.
Comment 17 Virgil Dicu [:virgil] [QA] 2013-02-07 06:27:56 PST
Based on comment 15 and the fact that there are no crashes in crash reports for the last 4 weeks I'm setting this to verified.

Note You need to log in before you can comment on or make changes to this bug.