Firefox 4.0b8 Crash Report [@ DEBUG_CheckWrapperThreadSafety(XPCWrappedNative const*) ]




JavaScript Engine
7 years ago
6 years ago


(Reporter: marcia, Assigned: Ben Turner (not reading bugmail, use the needinfo flag!))


({crash, topcrash})

crash, topcrash

Firefox Tracking Flags

(blocking2.0 final+)


(Whiteboard: softblocker, fixed-in-tracemonkey, crash signature)


(5 attachments)



7 years ago
Seen while reviewing early B8 crash stats. to the crashes so far which are all Windows.

Frame 	Module 	Signature [Expand] 	Source
0 	xul.dll 	DEBUG_CheckWrapperThreadSafety 	js/src/xpconnect/src/xpcwrappednative.cpp:3761
1 	xul.dll 	XPCCallContext::Init 	js/src/xpconnect/src/xpccallcontext.cpp:197
2 	xul.dll 	XPC_WN_NoHelper_Resolve 	js/src/xpconnect/src/xpcwrappednativejsops.cpp:750
3 	mozjs.dll 	CallResolveOp 	js/src/jsobj.cpp:4750
4 	mozjs.dll 	js_GetPropertyHelper 	js/src/jsobj.cpp:5255
5 	mozjs.dll 	js_GetMethod 	js/src/jsobj.cpp:5291

Comment 1

7 years ago
#3 topcrash on beta 8 as number of users starts to ramp up.

Comment 2

7 years ago
Currently this is the #1 crash on Beta 8. All seven comment appears to be in German. Here are the addon correlations:

100% (349/350) vs.   2% (379/21334) {d49175b3-3fd8-43b8-b28e-da5d47f3c398}
51% (177/350) vs.  12% (2617/21334) {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d} (Adblock Plus,
17% (58/350) vs.   2% (481/21334) {a0d7ccb3-214d-498b-b4aa-0e8fda9a7bf7} (WOT,
9% (33/350) vs.   1% (135/21334) (German Dictionary,
98% (344/350) vs.  91% (19382/21334) {972ce4c6-7e08-4474-a285-3208198ce6fd} (Default,
15% (51/350) vs.   9% (1856/21334) {CAFEEFAC-0016-0000-0022-ABCDEFFEDCBA}


7 years ago
blocking2.0: --- → ?

Comment 3

7 years ago
Anyone on this?  It would be a good one to get for beta9 to erase the regression thats blown up after the release of beta8.   It was intermittent on trunk first seen around nov 25th,  but now its a high volume, #2 two crash on beta8

date     total    breakdown by build
         crashes  count build, count build, ...
20101125 1 4.0b8pre2010112404 	1 , 
20101128 1 4.0b8pre2010112803 	1 , 
20101129 1 4.0b8pre2010112903 	1 , 
20101202 1 4.0b8pre2010120103 	1 , 
20101213 1 4.0b8pre2010121203 	1 , 
20101215 1 4.0b9pre2010121503 	1 , 
20101216 1 4.0b8pre2010121203 	1 , 
20101219 1 4.0b9pre2010121803 	1 , 
20101221 23 4.0b82010121417 	23 , 
20101222 109  	108 4.0b82010121417, 
        		1 4.0b9pre2010122203, 
20101223 220  	215 4.0b82010121417, 
        		4 4.0b9pre2010122203, 	1 4.0b9pre2010122303, 
20101224 242 4.0b82010121417 	242 , 
20101225 318  	317 4.0b82010121417, 
        		1 4.0b8pre2010112611, 
20101226 415  	411 4.0b82010121417, 
        		2 4.0b9pre2010122603, 	2 4.0b9pre2010122503, 
20101227 481 4.0b82010121417 	481 , 
20101228 439 4.0b82010121417 	439 ,


7 years ago
Keywords: topcrash
I enabled DEBUG_CheckWrapperThreadSafety to smoke out add-ons using JS across threads. Boris thought it would not recover and it seems he's right, or (need a debug build) perhaps it's just crashing due to some easy bug in the code.

But the upshot is that some add-on(s) use XPConnect unsafely across threads. We need to find and block them, and evangelize their authors to do something else.

Are all the crashes in the main thread?

Why can't we get a backtrace past js_GetMethod?

Assignee: general → brendan
Look for add-ons with binary components, since gal blocked JS thread manager thread-crossing usage.


Comment 7

7 years ago
running some queries to get a more comprehensive list, but here is a look at a sample of 75 crashes.  

Some of this might be easy to fix since the crashes are coming from lab addons.  The test pilot and NO_ADDONS_FOUND also need a closer look since test pilot ships with beta8.  It's either violating the thread safety rules, or possibly just around as an innocent bystander when the crash happens and the the addon list is gathered up and sent in with the crash report.  this could also be the case for others on the list.

once we get this labs addons sorted out it will be easier to spot other possible violators.

  75 total reports on this sample

  62 "">Test Pilot
  30 "">Adblock Plus
  10 "">Web of Trust - Safe Browsing Tool
   7 "">Microsoft .NET Framework Assistant
   4 "">Google Toolbar
   3 "">FlashGot
   2 "">Nightly Tester Tools
   2 "">PrintPDF
   2 "">Minimap Sidebar
   2 "">Speed Dial
   2 "">Xmarks Sync
   2 "">Session Manager
   2 "">Add-on Compatibility Reporter
   2 "">ReminderFox
   2 "">Easy YouTube Video Downloader
   1 "">
   1 "">gTranslate
   1 "">FoxTab
   1 "">LastPass Password Manager
   1 "">Read It Later
   1 "">Greasemonkey
   1 "">BetterPrivacy
   1 "">LavaFox V1-Green
   1 "">YouPlayer
   1 "">Cooliris
   1 "">AutoPager
   1 "">Tab Scope
   1 "">
   1 "">WebMail Notifier (for Gmail, Hotmail, Yahoo, AOL and so on)
   1 "">Element Hiding Helper for Adblock Plus
   1 "">Exif Viewer
   1 "">Mozilla Labs: Prospector - Instant Preview
   1 "">F1 by Mozilla Labs
   1 "">Mozilla Labs: Lab Kit
   1 "">Mozilla Labs: Prospector - Find Suggest
   1 "">Mozilla Labs: Prospector - Speak Words
   1 "">RSS Ticker
   1 "">BlackFox V1-Blue
   1 "">Showcase
   1 "">ColorfulTabs
   1 "">Facebook PhotoZoom
   1 "">Download YouTube Videos as MP4 and FLV
   1 "">IE Tab Plus (FF 3.6+)
   1 "">Personas Plus
   1 "">Firefox Sync

Comment 8

7 years ago
Created attachment 502259 [details]
inventory of addon info for all crashes with this signature on jan 07
I don't think TestPilot uses threads -- cc'ing Jono to confirm.

Cc'ing Jorge too. IIRC we had a list of add-ons with binary components. Would be great to cross-index against comment 7.

OS: Windows 7 → Windows XP

"">Microsoft .NET Framework Assistant

from attachment 502259 [details] (comment 8).

OS: Windows XP → All

Comment 11

7 years ago
Created attachment 502275 [details]
better list with addons not served off amo

ah, this could get harder.  several of these in the top slots are addons not served off amo, with no contact info.  we'll have to look at uuid's and try and figure out what they are, and who provides them, with some google searches.

Comment 12

7 years ago
Created attachment 502276 [details]
summary listing with counts of addons spotted on jan. 7
So...  In this case, we're consistently crashing at 0x0.  |wrapper| is known to be non-null (the callsite checked it, in fact, in the cases I looked at).  So either wrapper->mThread is null (how?) or somehow wrapper->mThread's vtable pointer is null.  

That's assuming we trust our line numbers and crash addresses, which I mostly do.

Note that mThread is set in XPCWrappedNative::FinishInit, which I would certainly hope has been checked by that point.
(In reply to comment #5)
> Why can't we get a backtrace past js_GetMethod?

I'm not really sure yet. I poked at it a little bit using this crash report:

Without having the binaries available locally, just the symbols, and loading it in WinDBG, the stack was even shorter than ours, so that's not helpful. After downloading the binaries, it's able to unwind all the way (huh, I wonder what it's doing smarter than us?) I'll attack a full stack in a minute.

Unrelated to all that, while poking this minidump in the debugger, I noticed that it has mzvkbd3.dll installed, which is some Kaspersky thing. chofmann: it might be worthwhile checking correlations on that.
Created attachment 502501 [details]
full stack

Comment 17

7 years ago
> it might be worthwhile checking correlations on that.
Correlations are automatically generated by dbaron's script.
For mzvkbd3.dll: 24% vs 4%.

May be related to German because all comments are in German.
Comments says (translated from Google):
"Crash-message always comes when you close Firefox beta 8"
"have only the windows shut with x and then this error message!"

Comment 18

7 years ago
(In reply to comment #15)
> Unrelated to all that, while poking this minidump in the debugger, I noticed
> that it has mzvkbd3.dll installed, which is some Kaspersky thing. chofmann: it
> might be worthwhile checking correlations on that.

I looked at a sample of 100 reports for jan 7.  mzvkbd3.dll is around about 32% of the time, and version distribution is broad, so if there is a thread safety problem there it may have been around since early versions.

  18 mzvkbd3.dll
   4 mzvkbd3.dll
   4 mzvkbd3.dll
   4 mzvkbd3.dll
   2 mzvkbd3.dll
   1 mzvkbd3.dll

Do we have a "Binary Addons - Thread Saftey Do's and Don't's" document around somewhere?   I have some contacts from Kaspersky that might be helpful in checking their code for problems.
blocking2.0: ? → final+
Whiteboard: softblocker
We do not want binary add-on components touching main-thread-only objects at all. Andreas: even without the DEBUG_* junk, do we have that checked now?


Comment 20

7 years ago
No, I don't think we check that in opt builds. JS access is checked, but not access from native code extensions.
(In reply to comment #20)
> No, I don't think we check that in opt builds. JS access is checked, but not
> access from native code extensions.

Any reason why we shouldn't check from C++ too?

Talked to jst and bent, bent is kindly taking this one.

Assignee: brendan → bent.mozilla
(In reply to comment #19)
> We do not want binary add-on components touching main-thread-only objects at
> all.

I meant from non-main threads, of course.

Created attachment 503266 [details] [diff] [review]
Patch, v1

So the stack in attachment 502501 [details] shows something pretty disturbing. We're doing a lot of stuff with XPCOM services after we've shut down the thread manager and the service manager.

I don't know how to prevent that in a simple or safe manner for Firefox 4. We need to take a long look at our shutdown process (specifically the interaction between the cycle collector and XPConnect and JS components that may be used as services) after we ship.

For now we can simply prevent the crash by switching away from using nsIThread[Manager] to simply using NSPR.

mThread is null in this case because the wrapper was created after the thread manager was shut down. After that time it will refuse to hand out anything for NS_GetCurrentThread and friends, even if we're on the main thread (as we are in the stack above).

NS_IsMainThread has been made to work even after the thread manager has been shut down, so we can continue to use that.
Attachment #503266 - Flags: review?(jst)

Comment 25

7 years ago
not many comments overall since the first of the year, but most are about shutting down, and just about all are in German for some reason.  here are comments not about shut down.

    1 How do I get this message now entlich away ????? This interferes with the times! :-(
    1 of the crash took place after the Schlie├čun (the second time).
    1 Hello, | crashed constantly on my Firefox 4.0 Beta 7th I have previously tried it with other versions but also the crashes. It is annoying my favorite browser but the constant crashes. | I was on the computerized images of bSeite
    1 you can get the button back to the old home place? (left). | printer I have to add every once and again.
    One oh that's not a good way to start

Comment 26

7 years ago
I hit this when I run chromebug1.7, exit, then run the same profile without -chromebug on the command line, the exit.

Note that Extensions tab is incorrect on that report.  Says none, but that is false.

I guess that the comment:
One oh that's not a good way to start
was when I first began to use 4.0b9pre trunk, its the only time I remember this hitting when I was not exiting.

I do not have C++ support on this machine.


7 years ago
Attachment #503266 - Flags: review?(jst) → review+
Whiteboard: softblocker → softblocker, fixed-in-tracemonkey
Last Resolved: 7 years ago
Resolution: --- → FIXED
Crash Signature: [@ DEBUG_CheckWrapperThreadSafety(XPCWrappedNative const*) ]
You need to log in before you can comment on or make changes to this bug.