Crashes reading native app preferences, mostly on startup, mostly on OS X 10.9 and 10.9.1, with thunder@xunlei.com extension

NEW
Assigned to

Status

()

P3
critical
5 years ago
11 months ago

People

(Reporter: smichaud, Assigned: smichaud)

Tracking

({reproducible, topcrash-mac})

27 Branch
All
Mac OS X
reproducible, topcrash-mac
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox28+ wontfix, firefox29+ wontfix, firefox30+ verified)

Details

(Whiteboard: [startup crash][STR in comment #31], crash signature)

Attachments

(5 attachments)

https://crash-stats.mozilla.com/report/list?signature=CoreFoundation%400x109881&product=Firefox&query_type=contains&range_unit=weeks&process_type=any&platform=mac&hang_type=any&date=2013-12-18+22%3A00%3A00&range_value=4#reports

These aren't really a Cocoa : Widgets bug.  They're most likely an Apple bug, but that's a convenient place to put them.

For now at least, these are mostly on the 27 branch, where they're the #4 Mac topcrasher.  But there are also a few on earlier and later branches.
(Assignee)

Updated

5 years ago
Crash Signature: [@ CoreFoundation@0x109881 ]
(Assignee)

Comment 1

5 years ago
These are mostly on OS X 10.9 and 10.9.1.  But there are also some on OS X 10.8 (CoreFoundation@0xd04c1) and 10.7 (CoreFoundation@0xe01d1).

I don't know if these are all related.  Even on the same OS, below the top the stacks can be quite different.

By the way, the assembly code for __HALT is just "int3" -- which halts in a debugger if one is present.
Crash Signature: [@ CoreFoundation@0x109881 ] → [@ CoreFoundation@0x109881 ] [@ CoreFoundation@0xd04c1 ] [@ CoreFoundation@0xe01d1 ]
(Assignee)

Updated

5 years ago
Summary: Crashes at __HALT (CoreFoundation) on OS X 10.9 and 10.9.1 → Crashes at __HALT (CoreFoundation), mostly on OS X 10.9 and 10.9.1
(Assignee)

Updated

5 years ago
Version: 29 Branch → 27 Branch
Here are some URLs in case this helps:

Total Count 	URL
4 	http://news.searchonme.com/?v=m1
2 	http://my.yahoo.com/
2 	http://ishare.iask.sina.com.cn/download.php?fileid=63132417
2 	about:blank
1 	http://www.macapp.cc/download.php?id=12469&did=1266&action=1385986692&from=autodesk-maya-2013
1 	https://mail.google.com/mail/u/0/?shva=1#inbox
1 	https://online.klarna.com/view_document.yaws?invno=513112593363620189&data=AZh2yUP4CIHdFPnwNVPNQpVozMUlKF9xcIYn0PTtjYd89vdC8cO8aGTmNWcppwg2bxTXDZPrLrpyTKWEoZ9CjLSE9heHvC1HtUYVC4S%2BEn4jtKRXcyfJerHugrgv5T7eD39eN%2BSVWBfMX4CJHQ%3D%3D
1 	http://web.cgi.weiyun.com/share_dl.fcg?sharekey=6ab05e33c632247150c704f956b1ab49&uin=131852&skey=&fid=34bf1d29-c745-4fc6-9fd3-e3e60f2b6da2&dir=&pdir=0c030200126fd2c29c7289604c0c174a&zn=iPhoneQQ_4.6.0.30_PP%E5%8A%A9%E6%89%8B%E5%85%8D%E8%B6%8A%E7%8B%B1%E7%8
1 	http://user.s6.qzone.qq.com/
1 	http://www.youtube.com/watch?v=1BqDtV1uktY&list=PLeYB07rUDOP5jAi6_n-aOzZZ6yK_sUI_7
1 	http://www.gutteruncensored.com/
1 	http://www.gamersky.com/ent/201312/312304_31.shtml
1 	https://news.google.com/nwshp?hl=zh-CN&tab=wn
1 	file:///Applications/MAMP/htdocs/portfolio/header.html
1 	https://mail.google.com/mail/u/0/?shva=1#inbox/
1 	http://d.pcs.baidu.com/file/8588547ecbabc957fa5fa8358d96fbbf?fid=2181285007-250528-151699923&time=1387368841&sign=FDTAER-DCb740ccc5511e5e8fedcff06b081203-IWUxggxFZRDLD6olKMHLRSTb1A4%3D&rt=sh&expires=1387369749&r=906220738&sh=1&sharesign=k6amqoHziyahR41rRe
1 	https://twitter.com/
1 	http://www.flickr.com/upload
1 	http://dupc.netadsopt.com/?s1=&s2=&s3=&s4=&s5=&d=px.pluginh
1 	ftp://ftp.mozilla.org/pub/firefox/candidates/27.0b2-candidates/build1/
1 	http://mail.qq.com/
(Assignee)

Comment 3

5 years ago
Here are the top few lines of the 10.9/10.9.1 crashes, translated using atos:

__HALT (in CoreFoundation) + 1
__CFBasicHashRehash (in CoreFoundation) + 2526
+[__NSArrayM __new:::::] (in CoreFoundation) + 274
__CFBasicHashAddValue (in CoreFoundation) + 86
CFDictionarySetValue (in CoreFoundation) + 217
objc_msgSend (in libobjc.dylib) + 0
+[CFPrefsSearchListSource withSuiteSearchListForIdentifier:user:locked:perform:] (in CoreFoundation) + 191
-[CFPrefsSearchListSource addSuiteSourceForIdentifier:user:] (in CoreFoundation) + 114
__60-[CFPrefsSearchListSource addSuiteSourceForIdentifier:user:]_block_invoke (in CoreFoundation) + 0
-[NSInvocation getArgument:atIndex:] (in CoreFoundation) + 196
__Block_byref_object_copy_ (in CoreFoundation) + 0
__Block_byref_object_dispose_ (in CoreFoundation) + 0
CFPreferencesCopyAppValue (in CoreFoundation) + 168
__CFPreferencesCopyAppValue_block_invoke (in CoreFoundation) + 0
(Assignee)

Comment 4

5 years ago
Apple's source for at least part of the CoreFoundation framework is available online, at http://opensource.apple.com/.  CFBasicHash.c (http://opensource.apple.com/source/CF/CF-855.11/CFBasicHash.c) uses HALT a lot for things like out-of-bounds parameters.
(Assignee)

Updated

5 years ago
Keywords: topcrash-mac
(Assignee)

Comment 5

5 years ago
The top few lines of the 27 and 28 branch crash stacks show something similar to what's going on in Mavericks:

__HALT (in CoreFoundation) + 1
__CFBasicHashRehash (in CoreFoundation) + 2185
__CFBasicHashAddValue (in CoreFoundation) + 75
CFDictionarySetValue (in CoreFoundation) + 189
+[CFPrefsSearchListSource withSuiteSearchListForIdentifier:user:locked:perform:] (in CoreFoundation) + 168
-[CFPrefsSearchListSource addSuiteSourceForIdentifier:user:] (in CoreFoundation) + 114
__60-[CFPrefsSearchListSource addSuiteSourceForIdentifier:user:]_block_invoke_0 (in CoreFoundation) + 0
+[CFPrefsSearchListSource withSearchListForIdentifier:perform:] (in CoreFoundation) + 366
__Block_byref_object_copy_ (in CoreFoundation) + 0
__Block_byref_object_dispose_ (in CoreFoundation) + 0
CFPreferencesCopyAppValue (in CoreFoundation) + 186
__CFPreferencesCopyAppValue_block_invoke_0 (in CoreFoundation) + 0
CFPreferencesGetAppIntegerValue (in CoreFoundation) + 34

(I've eliminated some calls to _cache_getImp, which are likely spurious.)
(Assignee)

Comment 6

5 years ago
(The previous comment concerned the crashes on OS X 10.8.X.)

Likewise with the crashes on OS X 10.7.5:

__HALT (in CoreFoundation) + 1
__CFBasicHashRehash (in CoreFoundation) + 2062
__CFBasicHashAddValue (in CoreFoundation) + 71
CFDictionarySetValue (in CoreFoundation) + 252
__CFXPreferencesGetSuiteSearchListForBundleIDAndUser (in CoreFoundation) + 112
__CFXPreferencesGetSearchListForBundleID (in CoreFoundation) + 645
___CFXPreferencesCopyAppValue_block_invoke_1 (in CoreFoundation) + 24
CFPreferencesCopyAppValue (in CoreFoundation) + 218
___CFXPreferencesCopyAppValue_block_invoke_1 (in CoreFoundation) + 0
(Assignee)

Comment 7

5 years ago
Interestingly, a lot of these (well over half) are startup crashes.

Not all of them, though.
Whiteboard: [startup crash]
(Assignee)

Comment 9

5 years ago
Crashes in WebKit, possibly related:

https://bugs.webkit.org/show_bug.cgi?id=81673
A 10.9.X variant.
Crash Signature: [@ CoreFoundation@0x109881 ] [@ CoreFoundation@0xd04c1 ] [@ CoreFoundation@0xe01d1 ] → [@ CoreFoundation@0x109881 ] [@ CoreFoundation@0x11da2 ] [@ CoreFoundation@0xd04c1 ] [@ CoreFoundation@0xe01d1 ]
(Assignee)

Updated

5 years ago
Summary: Crashes at __HALT (CoreFoundation), mostly on OS X 10.9 and 10.9.1 → Crashes reading native app preferences, mostly on startup, mostly on OS X 10.9 and 10.9.1
I looked by hand at about 10 reports for different versions of OS X.  All had the "thunder@xunlei.com" extension, which turns out to be a torrent client :-)

http://torrent-clients.findthebest.com/l/64/Xunlei-Thunder
Summary: Crashes reading native app preferences, mostly on startup, mostly on OS X 10.9 and 10.9.1 → Crashes reading native app preferences, mostly on startup, mostly on OS X 10.9 and 10.9.1, with thunder@xunlei.com extension
With the evidence we have so far (still not much), I'd guess these crashes are caused by invoking native preference APIs on an app before the OS has finished loading it.
迅雷 (Xunlei) is available from a Chinese site 迅雷看看 (http://www.kankan.com) in many versions, including a Mac one (Mac 迅雷).  It's commercial and closed source, though you can download a free beta from http://mac.xunlei.com.

The Xunlei Kankan site appears to be entirely in Chinese, and I'd guess that the Xunlei client is a "captive" one -- designed for exclusive use with that site.  So most of the people who see these bugs are probably Chinese people from China.  (The fact that the site uses simplified Chinese characters means, I assume, that it's not in Taiwan or Hongkong.)

For many years there's been a Japanese Bugzilla (http://bugzilla.mozilla.gr.jp/).  I wonder if there's a Chinese equivalent, and if this bug has been reported there?

Sorry to bug you, Gen, but I figure you might know, or at least know who to ask :-)
Flags: needinfo?(gen)
迅雷 means "thunder" :-)

Comment 15

5 years ago
Adding Jack as Xunlei is a Chinese site/service. Jack- could you help us find the right person in your team to help Steven with this bug?
Flags: needinfo?(gen)
(In reply to Steven Michaud from comment #11)
> I looked by hand at about 10 reports for different versions of OS X.  All
> had the "thunder@xunlei.com" extension, which turns out to be a torrent
> client :-)
> 
> http://torrent-clients.findthebest.com/l/64/Xunlei-Thunder

Just for the record, the same crash happens on Thunderbird, but without such an extension: 

https://crash-stats.mozilla.com/report/list?product=Thunderbird&query_search=signature&query_type=contains&reason_type=contains&date=2013-12-19&range_value=28&range_unit=days&hang_type=any&process_type=any&signature=CoreFoundation%400x109881

Of course, there are only two crashes in the past 28 days there, so the extension is definitely making things worse.

Comment 17

5 years ago
Hi,Steven Michaud
I am at the development team related to thunder@xunlei.com extension. Sorry for the problem maybe caused by the extension.
I should be most grateful if you would contact me through email and we can check to see what the problem is  and then handle it.
Let me know if there is anything I can do to help.
Thanks
(In reply to comment #17)

I have no more information than what's already here.

I'd hoped that Xunlei already knew about this crash, and might have fixed it.  Apparently not.

It's been a while since I opened this bug, so we should now have a lot more statistics on it.  I'll look through them to see what new information I can find (for example version numbers for the thunder@xunlei.com extension).  I should be able to do that in the next 2-3 days.
(In reply to comment #16)

These look unrelated.  Here's an atos translation of their top few lines:

__HALT (in CoreFoundation) + 1
_CFRuntimeCreateInstance (in CoreFoundation) + 532
____CFUUIDCreateWithBytesPrimitive_block_invoke (in CoreFoundation) + 0
_NARuntimeCreateInstance (in NetAuth) + 24
NAConnectionCreate (in NetAuth) + 204
kCFAllocatorDefault (in CoreFoundation) + 0
NAConnectToServerSync (in NetAuth) + 60
NetFSMountURLWithAuthenticationSync (in NetFS) + 56
netfs_MountURLWithAuthenticationSync (in NetFS) + 127
...
Here are some correlations for crashes with the signature CoreFoundation@0x109881 (on OS X 10.9.x):

97% (85/88) vs.  27% (755/2826) thunder@xunlei.com
32% (28/88) vs.  16% (457/2826) cpmanager@mozillaonline.com
32% (28/88) vs.  16% (461/2826) tabimprovelite@mozillaonline.com
30% (26/88) vs.  16% (440/2826) cehomepage@mozillaonline.com
27% (24/88) vs.  16% (441/2826) quicklaunch@mozillaonline.com
15% (13/88) vs.   4% (125/2826) HighSpeedDownload@dbank.com
24% (21/88) vs.  15% (418/2826) share_all_cn@mozillaonline.com
14% (12/88) vs.   6% (162/2826) netvideohunter@netvideohunter.com (NetVideoHunter Video Downloader, https://addons.mozilla.org/addon/7447)
10% (9/88) vs.   4% (103/2826) {e4a8a97b-f2ed-450b-b12d-ee082ba24781} (Greasemonkey, https://addons.mozilla.org/addon/748)
19% (17/88) vs.  13% (360/2826) livemargins@mozillaonline.com
8% (7/88) vs.   2% (55/2826) foxmarks@kei.com (Xmarks (formerly Foxmarks), https://addons.mozilla.org/addon/2410)
10% (9/88) vs.   5% (136/2826) easyscreenshot@mozillaonline.com
7% (6/88) vs.   2% (50/2826) isreaditlater@ideashower.com (Read It Later, https://addons.mozilla.org/addon/7661)

By far the strongest correlation is with the thunder@xunlei.com extension, but there are also weaker correlations with other extensions.
Some of these actually have the translated __HALT signature (mostly on OS X 10.6.X).

https://crash-stats.mozilla.com/report/list?signature=__HALT&product=Firefox&query_type=contains&range_unit=weeks&process_type=any&platform=mac&hang_type=any&date=2014-02-21+21%3A00%3A00&range_value=4#reports
Crash Signature: [@ CoreFoundation@0x109881 ] [@ CoreFoundation@0x11da2 ] [@ CoreFoundation@0xd04c1 ] [@ CoreFoundation@0xe01d1 ] → [@ CoreFoundation@0x109881 ] [@ CoreFoundation@0x11da2 ] [@ CoreFoundation@0xd04c1 ] [@ CoreFoundation@0xe01d1 ] [@ __HALT ]
> I'll look through them to see what new information I can find (for
> example version numbers for the thunder@xunlei.com extension).

Our crash server doesn't let us collect this information
systemmatically.  But looking through the thunder@xunlei.com version
numbers, I saw mostly 2.0.2, 2.0.4 and 2.0.6.  I never saw a version
number higher than 2.0.6.

sam green, what is the current version number?
sam green:

What evidence we have (not much) suggests that these crashes aren't *caused* by the thunder@xunlei.com.  Though clearly that extension's presence makes them much more likely.

What does the thunder@xunlei.com extension do at Firefox startup?  If we knew, it might be easier to figure out this bug.  (I ask because these crashes mostly happen as Firefox is starting up.)

Even better if we knew how to reproduce these crashes.  Clearly a lot of your customers see them.  Does your company have a bug reporting system?  And if it does, have these crashes been reported there?
More clues as to what's going on here.

The following two links suggest that these crashes can happen when a component (an app component or a system component) has been partially uninstalled, or was incorrectly installed:

http://lists.apple.com/archives/carbon-dev/2011/Feb/msg00014.html
https://discussions.apple.com/thread/1406939

Comment 25

5 years ago
(In reply to Steven Michaud from comment #23)
> sam green:
> 
> What evidence we have (not much) suggests that these crashes aren't *caused*
> by the thunder@xunlei.com.  Though clearly that extension's presence makes
> them much more likely.
> 
> What does the thunder@xunlei.com extension do at Firefox startup?  If we
> knew, it might be easier to figure out this bug.  (I ask because these
> crashes mostly happen as Firefox is starting up.)
> 
> Even better if we knew how to reproduce these crashes.  Clearly a lot of
> your customers see them.  Does your company have a bug reporting system? 
> And if it does, have these crashes been reported there?

Hi, Steven Michaud
Thanks for your information.
The current version number of thunder@xunlei.com is 2.0.6.
I have found that on firefox 26 this extension works ok. But if I upgraded firefox to version 27,firefox crashed at startup in OSX 10.9, but ok in OSX 10.8. Please have a look at this case.
thunder@xunlei.com is a common extension. When a page is loaded, it will insert an embed in the page,the type of embed is application/thunder_download_plugin. If you click a link which is a http or ftp file, the extension will use ThunderPlugIn to invoke thunder which is an app to download the file.
The ThunderPlugIn is located in extensions/thunder@xunlei.com/components
Thanks for the information.  Here's one more question I forgot to ask:

Where can I download the current version of the thunder@xunlei.com extension?

I've found I can download a beta version at http://mac.xunlei.com/ (specifically at http://down.sandai.net/mac/thunder_dl2.3.0.1280_Beta.dmg).  But I can't find how to download the current version.  (I know some Chinese, but I'm not very comfortable navigating the http://www.kankan.com/ web site.)
(Assignee)

Updated

5 years ago
Crash Signature: [@ CoreFoundation@0x109881 ] [@ CoreFoundation@0x11da2 ] [@ CoreFoundation@0xd04c1 ] [@ CoreFoundation@0xe01d1 ] [@ __HALT ] → [@ CoreFoundation@0x109881 ] [@ CoreFoundation@0x11da2 ] [@ CoreFoundation@0xd04c1 ] [@ CoreFoundation@0xe01d1 ] [@ __HALT ] [@ libobjc.A.dylib@0xb310 ] [@ objc_msgSend_vtable12 | CoreFoundation@0xfa92 ]

Comment 27

5 years ago
(In reply to Steven Michaud from comment #26)
> Thanks for the information.  Here's one more question I forgot to ask:
> 
> Where can I download the current version of the thunder@xunlei.com extension?
> 
> I've found I can download a beta version at http://mac.xunlei.com/
> (specifically at http://down.sandai.net/mac/thunder_dl2.3.0.1280_Beta.dmg). 
> But I can't find how to download the current version.  (I know some Chinese,
> but I'm not very comfortable navigating the http://www.kankan.com/ web site.)
Yes,you can download a beta version at http://mac.xunlei.com(specifically at http://down.sandai.net/mac/thunder_dl2.3.0.1280_Beta.dmg).After installed the app, you can find thunder@xunlei.com extension at /Applications/Thunder.app/Contents/BrowserPlugins/Thunder\ Extension.safariextz
This crash is now ranked #7 in the Firefox 28 Mac-specific crashes and has risen 21 positions in the last week.
status-firefox28: --- → affected
tracking-firefox28: --- → ?
(Assignee)

Updated

5 years ago
Crash Signature: [@ CoreFoundation@0x109881 ] [@ CoreFoundation@0x11da2 ] [@ CoreFoundation@0xd04c1 ] [@ CoreFoundation@0xe01d1 ] [@ __HALT ] [@ libobjc.A.dylib@0xb310 ] [@ objc_msgSend_vtable12 | CoreFoundation@0xfa92 ] → [@ CoreFoundation@0x109881 ] [@ CoreFoundation@0x11da2 ] [@ CoreFoundation@0xd04c1 ] [@ CoreFoundation@0xe01d1 ] [@ __HALT ] [@ libobjc.A.dylib@0xb310 ] [@ objc_msgSend_vtable12 | CoreFoundation@0xfa92 ] [@ libobjc.A.dylib@0x5097 ]
Actually, with all the signatures together this is probably the #3 or #2 Mac topcrash on the 28 branch.
Translation (using atos) of top few lines of crash stacks with most recently discovered signature (which also has the most crashes):

libobjc.A.dylib@0x5097

objc_msgSend (in libobjc.dylib) + 23
__CFBasicHashRehash (in CoreFoundation) + 2526
+[__NSArrayM __new:::::] (in CoreFoundation) + 274
__CFBasicHashAddValue (in CoreFoundation) + 86
CFDictionarySetValue (in CoreFoundation) + 217
objc_msgSend (in libobjc.dylib) + 0
+[CFPrefsSearchListSource withSuiteSearchListForIdentifier:user:locked:perform:] (in CoreFoundation) + 191
-[CFPrefsSearchListSource addSuiteSourceForIdentifier:user:] (in CoreFoundation) + 114
__60-[CFPrefsSearchListSource addSuiteSourceForIdentifier:user:]_block_invoke (in CoreFoundation) + 0
+[CFPrefsSearchListSource withSearchListForIdentifier:perform:] (in CoreFoundation) + 388
__Block_byref_object_copy_ (in CoreFoundation) + 0
__Block_byref_object_dispose_ (in CoreFoundation) + 0
CFPreferencesCopyAppValue (in CoreFoundation) + 168
__CFPreferencesCopyAppValue_block_invoke (in CoreFoundation) + 0
...
OK, I can reproduce these crashes!  Even with today's mozilla-central nightly!  I tested on OS X 10.9.

STR

1) Visit http://down.sandai.net/mac/thunder_dl2.4.0.1356_Beta.dmg and download the file.
2) Mount the DMG and click the 同意 (OK) button in the dialog's lower right.
3) Drag the Thunder app to /Applications, then run it once and quit it (退出).
4) Run FF, chose File : Open File and navigate to
   /Applications/Thunder.app/Contents/BrowserPlugins/ThunderExtensionForFirefox.xpi.
5) Allow the extension to install itself and then restart.
6) Navigate to ftp://ftp.mozilla.org/pub/ ... and crash.

With FF 27.0.1 you generally crash on startup (after having downloaded at least one file using the xunlei@thunder.com extension and Thunder app).
Keywords: reproducible
Whiteboard: [startup crash] → [startup crash][STR in comment #31]
Since this is a topcrasher and reproducible, I'll now make it my top priority to figure out why this bug happens.
Assignee: nobody → smichaud
Awesome, thanks Steven!

Ioana, can you have someone on your team attempt to reproduce Steven's steps in comment 31? At least we'll know that we can verify it once it's fixed.
Flags: needinfo?(ioana.budnar)
Despite what I said in comment #23 and comment #24, I now think it's very likely these crashes are triggered by a bug in the thunder@xunlei.com extension.  The bug happens with the very first installation (of both the Thunder app and Firefox), so clearly it isn't an issue with reinstallation, deinstallation or upgrading.
Created attachment 8383278 [details]
lldb crash stack
I successfully reproduced the crash 5 times out of 5 tries using latest Nightly and Firefox 28 beta 7 so the STR are 100% reliable.

bp-00cec6c8-df47-489a-92e3-e6d4d2140228
bp-a6097fa6-2549-4663-b0e5-0838d2140228
bp-86d73548-4c6d-434f-b224-e91cf2140228
bp-46cd88d3-001d-425e-9848-3dede2140228
bp-f4a2208b-080a-4caf-be0c-80e9b2140228
Flags: needinfo?(ioana.budnar)
I did my testing using Mac OS X 10.9.1.
(In reply to Bogdan Maris, QA [:bogdan_maris] from comment #36)
> STR are 100% reliable.

Great, thanks Bogdan.
Comment hidden (obsolete)
This bug is very strange, but I'm slowly making progress.

The ThunderPlugIn (loaded into the main process by thunder@xunlei.com) calls CFPreferencesGetAppBooleanValue() twice (from ThunderPlugIn`is_browser_detect_enabled).  If either call is allowed to go through unchanged, crashes eventually happen.  The crashes don't happen if these calls are changed in ways I describe below.

Both calls have the applicationID parameter set to "com.xunlei.Thunder".  The key parameter is set to either "DetectIE" or "UndetectWithCommandKey".  Neither of these settings exists on my system, and CFPreferencesGetAppBooleanValue() returns 'false' for both of them.

The CoreFoundation framework has an internal (non-exported) _SEARCH_LISTS symbol that's used when searching for an application preference.  It's a CFDictionaryRef, and somehow it gets corrupted as an indirect result of the two calls to CFPreferencesGetAppBooleanValue() mentioned above.  This bug's crashes happen trying to access that CFDictionaryRef.  But _SEARCH_LISTS is still uncorrupt after the second of the ThunderPlugIn's two calls to CFPreferencesGetAppBooleanValue() returns.

The crashes don't happen if you change the applicationID for those two calls to anything else, even an arbitrary value (like "com.blah.blah").  They also don't happen if (in your hook for CFPreferencesGetAppBooleanValue()) you return "false", without ever having called the CoreFoundation method.  They do still happen if you just change the key to an arbitrary value (like "blah").

I'll keep digging, but for now this looks like some combination of a Thunder bug and an OS bug.
I should have mentioned that both calls to CFPreferencesGetAppBooleanValue() from the ThunderPlugIn happen on the main thread (as do most but not all of the crashes).
Created attachment 8386461 [details]
Trace of ThunderPlugIn's calls to CFPreferencesGetAppBooleanValue()

The top three lines of this stack show the artificial hang I had to induce to be able to get the stack at all.
An extra call to CFRetain() on _SEARCH_LISTS doesn't stop the crashes.  So it's not that _SEARCH_LISTS is getting prematurely deleted.
Interestingly, though my trace from comment #42 shows ThunderPlugIn being called on the main thread in the main process, none of my crash stacks show ThunderPlugIn in the list of modules.  Could it be that these crashes are caused by code that deliberately unloads the plugin?

sam green, do you have anything to say about this?

Could *we* be doing this?  I really don't think so.  But if anyone here knows better, please correct me.
Flags: needinfo?(shenyixin99)
Tomorrow I'll try to find what code unloads the ThunderPlugIn -- presuming that's what's happening.
We can no longer take any speculative fixes on FF28, wontfixing on that version and we'll hope Steven can find something that can help with 29 and later.
status-firefox28: affected → wontfix
status-firefox29: --- → affected
status-firefox30: --- → affected
tracking-firefox28: ? → +
tracking-firefox29: --- → +
tracking-firefox30: --- → +
I've now figured this bug out.  It's basically an Apple bug, or more precisely a design flaw.  But it's buried so deep in Apple system code (and has been for so long) that it's extremely unlikely that Apple will ever fix it.

So someone needs to around it.  For various reasons I think this "someone" should be Xunlei.  I will suggest an easy workaround in a later comment.

It's the Xunlei ThunderPlugIn that triggers the Apple bug.  And it does it so systematically in FF 27 and up that its presence (via the thunder@xunlei.com extension) makes those versions of Firefox unusable.  It's true that the bug doesn't get triggered (and the crashes don't happen) in FF 26 and below, but (as I'll explain below) that's because the ThunderPlugIn's behavior is different in FF 26 and below -- not because those earlier versions of Firefox behaved differently.

It will be very easy for Xunlei to work around this bug -- all that's needed (as I'll explain below) are a couple small changes to the ThunderPlugIn's code.  It would be much more difficult for Firefox to work around it:  Essentially we'd need to hook the ThunderPlugIn's calls to CFPreferencesGetAppBooleanValue().  And this is made even more difficult by the fact that Firefox doesn't load the ThunderPlugIn directly -- it's the thunder@xunlei.com extension that does this.

In retrospect, these crashes have been effecting Firefox and other apps for years, usually in small numbers.  But I don't think Firefox itself has ever been the trigger -- instead it's been modules loaded and unloaded by Firefox (directly or indirectly).

I'll go into more detail in subsequent comments.
Created attachment 8387091 [details]
Trace of call to unload ThunderPlugIn

It turns out we *are* unloading the ThunderPlugIn, as this stack shows.  But we're doing nothing wrong -- we're doing what's appropriate to the way the thunder@xunlei.com extension loads the ThunderPlugIn, as you'll see in my next comment.
Created attachment 8387097 [details]
thunder@xunlei.com's chrome/common.js

Here's the thunder@xunlei.com code that loads and unloads the ThunderPlugIn.

The following two lines in the ThunderDylib().__construct load it:

  var dylibPath = profileDir +
                  "/extensions/thunder@xunlei.com/components/ThunderPlugIn" ;
  ctypesDylib = ctypes.open(dylibPath);

And the following line in the ThunderDylib().free function unloads it:

  ctypesDylib.close();
As I mentioned above, this is basically an Apple bug.  Here's a detailed explanation.

As I also mentioned above (comment #40), the CoreFoundation framework has an internal CFDictionaryRef object that's used when searching for application preferences (using the CFPreferences API https://developer.apple.com/library/mac/documentation/CoreFoundation/Reference/CFPreferencesUtils/Reference/reference.html#//apple_ref/doc/uid/20001450-CH201-F15087).  On OS X 10.8 and up it's called SEARCH_LISTS.  On OS X 10.7 it's called __sourceListCache.

These dictionaries' "keys" are string objects that look like this:

"com.xunlei.Thunder"
"suite/com.xunlei.Thunder/kCFPreferencesAnyUser"
"suite/com.xunlei.Thunder/kCFPreferencesCurrentUser"

Their "values" are CFPrefsSearchListSource or CFXPreferencesSearchListSource objects used (presumably) to do searches specific to their "keys".

The "keys" are constructed using CFStringCreateWithFormat(), ultimately from the parameters passed in to methods like CFPreferencesGetAppBooleanValue().  The standard way to "construct" a CFStringRef "constant" is to do e.g. CFSTR("com.xunlei.Thunder"), which creates a reference to data stored in the __cfstring section of the __DATA segment.  This is very efficient, because no matter how many times you do CFSTR("com.xunlei.Thunder"), they will all refer to a single object in the __cfstring section.  This is in fact what the ThunderPlugIn does when it calls CFPreferencesGetAppBooleanValue().  But that single object in the ThunderPlugIn's __cfstring section goes out of scope when the ThunderPlugIn gets unloaded.

CFStringCreateWithFormat() isn't particularly well documented.  But apparently it *doesn't* copy the contents of CFStringRef objects that are passed to it.  So the "keys" it creates for the SEARCH_LISTS dictionary contain references to the contents of the CFStringRef objects that were passed to methods like CFPreferencesGetAppBooleanValue() -- particularly to the applicationID parameters (like "com.xunlei.Thunder").  So if the contents of any of those applicationID parameters goes out of scope, the SEARCH_LISTS dictionary will become unusable -- any attempt to iterate over its keys will cause crashes.

Apple's bug is that is *should not* have used externally-provided data internally.  It should have *copied* that data, not *referenced* it.  If it had, unloading the ThunderPlugIn wouldn't have triggered these crashes.
The ThunderPlugIn makes the following two calls to CFPreferencesGetAppBooleanValue():

CFPreferencesGetAppBooleanValue(CFSTR("DetectIE"), CFSTR("com.xunlei.Thunder"), &exists)
from is_browser_detect_enabled

CFPreferencesGetAppBooleanValue(CFSTR("UndetectWithCommandKey"), CFSTR("com.xunlei.Thunder"), &exists)
from is_command_key_undetect_enable

The simplest way to work around this bug is to replace each instance of 'CFSTR("com.xunlei.Thunder")' with 'CFStringCreateWithCString(kCFAllocatorDefault, "com.xunlei.Thunder", kCFStringEncodingUTF8)'.  I've tried this (in an interpose library that hooks ThunderPlugIn's calls to CFPreferencesGetAppBooleanValue()) and it works perfectly.

sam green, please try this and report back with your results.
The ThunderPlugIn doesn't call CFPreferencesGetAppBooleanValue() when loaded in FF 26 or below, or any other CFPreferences method.  This is why these crashes don't happen (with the thunder@xunlei.com extension) on FF 26 and earlier.
Created attachment 8387150 [details]
Interpose library for debugging

Here's the interpose library I used to figure out this bug.  It contains instructions for building and use.

It also contains a module_dlsym() method I wrote that can find non-exported symbols :-)  Like _SEARCH_LISTS and ___sourceListCache.
(Assignee)

Updated

5 years ago
Crash Signature: [@ CoreFoundation@0x109881 ] [@ CoreFoundation@0x11da2 ] [@ CoreFoundation@0xd04c1 ] [@ CoreFoundation@0xe01d1 ] [@ __HALT ] [@ libobjc.A.dylib@0xb310 ] [@ objc_msgSend_vtable12 | CoreFoundation@0xfa92 ] [@ libobjc.A.dylib@0x5097 ] → [@ CoreFoundation@0x109881 ] [@ CoreFoundation@0x11da2 ] [@ CoreFoundation@0xd04c1 ] [@ CoreFoundation@0xe01d1 ] [@ CoreFoundation@0x1096c1 ] [@ __HALT ] [@ libobjc.A.dylib@0xb310 ] [@ objc_msgSend_vtable12 | CoreFoundation@0xfa92 ] [@ libobjc.A.d…

Comment 54

5 years ago
(In reply to Steven Michaud from comment #51)
> The ThunderPlugIn makes the following two calls to
> CFPreferencesGetAppBooleanValue():
> 
> CFPreferencesGetAppBooleanValue(CFSTR("DetectIE"),
> CFSTR("com.xunlei.Thunder"), &exists)
> from is_browser_detect_enabled
> 
> CFPreferencesGetAppBooleanValue(CFSTR("UndetectWithCommandKey"),
> CFSTR("com.xunlei.Thunder"), &exists)
> from is_command_key_undetect_enable
> 
> The simplest way to work around this bug is to replace each instance of
> 'CFSTR("com.xunlei.Thunder")' with
> 'CFStringCreateWithCString(kCFAllocatorDefault, "com.xunlei.Thunder",
> kCFStringEncodingUTF8)'.  I've tried this (in an interpose library that
> hooks ThunderPlugIn's calls to CFPreferencesGetAppBooleanValue()) and it
> works perfectly.
> 
> sam green, please try this and report back with your results.

Hi,Steven Michaud, it works. We use the same plugin at safari, but it did not cause a crash and works well。We will upgrade the firefox plugin next version. Thank you very much.(In reply to Steven Michaud from comment #50)
> As I mentioned above, this is basically an Apple bug.  Here's a detailed
> explanation.
> 
> As I also mentioned above (comment #40), the CoreFoundation framework has an
> internal CFDictionaryRef object that's used when searching for application
> preferences (using the CFPreferences API
> https://developer.apple.com/library/mac/documentation/CoreFoundation/
> Reference/CFPreferencesUtils/Reference/reference.html#//apple_ref/doc/uid/
> 20001450-CH201-F15087).  On OS X 10.8 and up it's called SEARCH_LISTS.  On
> OS X 10.7 it's called __sourceListCache.
> 
> These dictionaries' "keys" are string objects that look like this:
> 
> "com.xunlei.Thunder"
> "suite/com.xunlei.Thunder/kCFPreferencesAnyUser"
> "suite/com.xunlei.Thunder/kCFPreferencesCurrentUser"
> 
> Their "values" are CFPrefsSearchListSource or CFXPreferencesSearchListSource
> objects used (presumably) to do searches specific to their "keys".
> 
> The "keys" are constructed using CFStringCreateWithFormat(), ultimately from
> the parameters passed in to methods like CFPreferencesGetAppBooleanValue(). 
> The standard way to "construct" a CFStringRef "constant" is to do e.g.
> CFSTR("com.xunlei.Thunder"), which creates a reference to data stored in the
> __cfstring section of the __DATA segment.  This is very efficient, because
> no matter how many times you do CFSTR("com.xunlei.Thunder"), they will all
> refer to a single object in the __cfstring section.  This is in fact what
> the ThunderPlugIn does when it calls CFPreferencesGetAppBooleanValue().  But
> that single object in the ThunderPlugIn's __cfstring section goes out of
> scope when the ThunderPlugIn gets unloaded.
> 
> CFStringCreateWithFormat() isn't particularly well documented.  But
> apparently it *doesn't* copy the contents of CFStringRef objects that are
> passed to it.  So the "keys" it creates for the SEARCH_LISTS dictionary
> contain references to the contents of the CFStringRef objects that were
> passed to methods like CFPreferencesGetAppBooleanValue() -- particularly to
> the applicationID parameters (like "com.xunlei.Thunder").  So if the
> contents of any of those applicationID parameters goes out of scope, the
> SEARCH_LISTS dictionary will become unusable -- any attempt to iterate over
> its keys will cause crashes.
> 
> Apple's bug is that is *should not* have used externally-provided data
> internally.  It should have *copied* that data, not *referenced* it.  If it
> had, unloading the ThunderPlugIn wouldn't have triggered these crashes.

It is weird , We use the same plugin at safari, it did not cause a crash and works well, but crash in firefox version 27
Flags: needinfo?(shenyixin99)

Comment 55

5 years ago
(In reply to Steven Michaud from comment #51)
> The ThunderPlugIn makes the following two calls to
> CFPreferencesGetAppBooleanValue():
> 
> CFPreferencesGetAppBooleanValue(CFSTR("DetectIE"),
> CFSTR("com.xunlei.Thunder"), &exists)
> from is_browser_detect_enabled
> 
> CFPreferencesGetAppBooleanValue(CFSTR("UndetectWithCommandKey"),
> CFSTR("com.xunlei.Thunder"), &exists)
> from is_command_key_undetect_enable
> 
> The simplest way to work around this bug is to replace each instance of
> 'CFSTR("com.xunlei.Thunder")' with
> 'CFStringCreateWithCString(kCFAllocatorDefault, "com.xunlei.Thunder",
> kCFStringEncodingUTF8)'.  I've tried this (in an interpose library that
> hooks ThunderPlugIn's calls to CFPreferencesGetAppBooleanValue()) and it
> works perfectly.
> 
> sam green, please try this and report back with your results.


Hi,Steven Michaud, I have try this and it works. We will upgrade the firefox plugin next version if testing is no problem.
Thank you very much.
(In reply to comment #54)

> It is weird , We use the same plugin at safari, it did not cause a
> crash and works well, but crash in firefox version 27

This is because the Xunlei Safari extension behaves differently.

The Xunlei Firefox extension loads the ThunderPlugIn (into the main
process) on Firefox startup and calls is_browser_detect_enabled() and
is_command_key_undetect_enable() in it -- which triggers the calls to
CFPreferencesGetAppBooleanValue() mentioned in comment #51.  The
extention then immediately unloads ThunderPlugIn, which triggers
Apple's bug.

The Xunlei Safari extension never explicitly loads ThunderPlugIn.
Instead it waits until the browser has loaded it (into a separate
plugin process), then calls is_browser_detect_enabled() and
is_command_key_undetect_enable() in it.  This way ThunderPlugIn never
gets unloaded (before the plugin process itself dies), and Apple's bug
never gets triggered.
(In reply to comment #55)

> Hi,Steven Michaud, I have try this and it works. We will upgrade the
> firefox plugin next version if testing is no problem.

I'm glad to hear it.  When will this next version be released?  Please
make it very soon!  The current version of the FF Xunlei extension
doesn't work at all in FF 27 and up, and also makes those Firefox
versions unusable.

> Thank you very much.

You're most welcome.
(Assignee)

Updated

5 years ago
Flags: needinfo?(shenyixin99)

Comment 58

5 years ago
Hi,Steven Michaud
We have fixed the crash and released the new client version at mac.xunlei.com
Thanks
Flags: needinfo?(shenyixin99)
Thanks, sam green, for the fix!

I downloaded thunder_dl2.5.0.1408_Beta.dmg (the latest beta) from http://mac.xunlei.com/ and tested with it (in FF 27 on OS X 10.9).  The STR from comment #31 no longer crash, so the bug does seem to be fixed (i.e. worked around).

I copied Thunder.app to /Applications, then ran Firefox and did the following to install the thunder@xunlei.com extension:

1) Choose File : Open File and browse to /Applications/Thunder.app/Contents/BrowserPlugins/ThunderExtensionForFirefox.xpi.
2) Click Open and allow the extension to be installed.
3) Restart Firefox when prompted.

When will this beta become the "standard" release version?  Once again I hope it is very soon.

We plan to start blocking older versions as soon as possible.  Since the extension is what Firefox "sees", we will block older versions of the extension.

Very fortunately you changed the thunder@xunlei.com extension's version in this latest download (the 2.5.0.1408_Beta one) to 2.0.7.  So we will be blocking version 2.0.6 and earlier.

There are two ways that Firefox uses to get a list of currently blocked extensions -- from a Mozilla server and from a list that's included with the Firefox distro (and the updater that's used to update the distro).  Because most of this bug's crashes happen on startup, we'll probably need to block in both places.
(Assignee)

Updated

5 years ago
Flags: needinfo?(shenyixin99)

Comment 60

4 years ago
Shuold this bug be morphed into a blocklisting bug, or are you going to file a separate one?
We'll use the steps in comment #31 and #59 to test this once the blocklist is staged, using the instructions here: https://wiki.mozilla.org/Blocklisting/Testing
Flags: needinfo?(smichaud)
Note that this is a startup crash so we might only be able to fix new installs using the blocklist that comes with Firefox vs. the one that is served up.
I've opened bug 988490.  Use it or not as you choose.

(In reply to comment #61)

What you say makes sense.  If I haven't answered all your questions, please needinfo me again.
Flags: needinfo?(smichaud)

Updated

4 years ago
Depends on: 988490
Setting back to 'affected' for 28, if we can push a blocklist fix without a respin, that would be ideal here.
status-firefox28: wontfix → affected
(Following up comment #59)

> When will this beta become the "standard" release version?  Once again I hope it is very soon.

sam green:  Is "Xunlei for Mac" only available in beta?  Please let us know, one way or the other.
Steven, any chance that this bug could be fixed for 29? We are going to release beta7 next Thursday. Thanks
Flags: needinfo?(smichaud)
Sylvestre, this bug is "fixed" by new versions of the thunder@xunlei.com extension that work around Apple's bug.  We plan to block versions of the extension < 2.0.7 on the Mac, but have had trouble implementing the block -- see bug 988490.  I don't know much about how blocking works, so I don't know what the trouble is.  But I strongly suspect it's "something stupid", which should be easy (and quick) to resolve.
Flags: needinfo?(smichaud)
(Following up comment #67)

Turns out there was nothing wrong with how the block was implemented.  You just have to wait at least 10 minutes (after FF startup) for it to have any effect ... which I didn't realize.

But this means that it will take a while for these crashes to disappear, or even to lessen significantly.  Since most of the crashes happen on startup, realistically they won't be "fixed" until most users download an FF distro whose built-in blocklist contains a block for thunder@xunlei.com < 2.0.7, using a different browser (not Firefox).
Comment hidden (obsolete)
(Assignee)

Updated

4 years ago
Depends on: 639524

Comment 70

4 years ago
(In reply to Steven Michaud from comment #68)
> Turns out there was nothing wrong with how the block was implemented.  You
> just have to wait at least 10 minutes (after FF startup) for it to have any
> effect ... which I didn't realize.

In an existing profile, you need to wait until we have a chance to download the new blocklist, which we only do once a day on idle, AFAIK. The built-in blocklist should be updated by automated checkins, but it only applies to new profiles (or ones that don't have a downloaded blocklit yet), see bug 639524, which you set in the dependencies anyhow.
Bug 639524 has been landed to FF30 so let's confirm this is fixed there - it's too late and too risky to throw it to FF29 release so we'll have to take the hit here for one more cycle.
status-firefox28: affected → wontfix
status-firefox29: affected → wontfix
Flags: needinfo?(jbecerra)
Flags: needinfo?(jbecerra)
Keywords: qawanted
I'm not sure how to verify the correct blocklist is being observed.  David, have you got any suggestions?
Flags: needinfo?(dmajor)
(In reply to [:tracy] Tracy Walker - QA Mentor from comment #72)
> I'm not sure how to verify the correct blocklist is being observed.  David,
> have you got any suggestions?

I work with the DLL blocklist which is a different beast with an unfortunately similar name. Kairo or Jorge would know more about the addon blocklist.

Although, from looking over the bug, it sounds like verifying that comment 31 doesn't crash would be sufficient to prove that the updated blocklist is in place.
Flags: needinfo?(dmajor)
Thanks David.  I verified on Fx30beta5 that STR's in comment #31 do not crash the browser.

also, the needinfo no longer seems relevant.
status-firefox30: affected → verified
Flags: needinfo?(shenyixin99)
Keywords: qawanted
Verifying that the blocklist "works" for this bug is *not* easy, for the following reasons:

1) Even testing with a fresh profile, Firefox doesn't check for blocklist updates until about 10 minutes after it's been started/restarted.
2) This bug's crashes often happen on startup.

I personally have been able to verify that the blocklist "works", but only because, in my test setup, the crashes don't happen if I start Firefox and let it sit (without doing anything at all).  See bug 988490 comment #20 and bug 988490 comment #21.

If all else fails, I'd be willing to VERIFY that myself, on the strength of what I just mentioned.

But I notice that Tracy has already done this :-)
This extension appears to be block listed as it is disabled by default.  enable it and Firefox crashes on restart.  

I didn't wait any particular length of time after installing the extension. It was just blocked.
By the way, FF 30 doesn't crash, even on startup, because bug 988490's block is already in its "built in" blocklist.

Updated

11 months ago
Priority: -- → P3
You need to log in before you can comment on or make changes to this bug.