Open Bug 675260 Opened 8 years ago Updated Last year

crash in DefaultFreeEntry coming from ShutdownNSS

Categories

(Core :: Security: PSM, defect, P3, critical)

5 Branch
defect

Tracking

()

Tracking Status
firefox6 + ---
firefox15 + ---
firefox16 + wontfix
firefox17 --- affected
firefox18 - ---

People

(Reporter: kairo, Unassigned)

References

Details

(Keywords: crash, regression, steps-wanted, Whiteboard: [psm-backlog])

Crash Data

This bug was filed from the Socorro interface and is 
report bp-75cd558a-aa19-44d9-8d85-839c22110729 .
============================================================= 

0 	mozcrt19.dll 	arena_dalloc_small 	obj-firefox/memory/jemalloc/crtsrc/jemalloc.c:4045
1 	mozcrt19.dll 	arena_dalloc 	obj-firefox/memory/jemalloc/crtsrc/jemalloc.c:4173
2 	mozcrt19.dll 	free 	obj-firefox/memory/jemalloc/crtsrc/jemalloc.c:6037
3 	plds4.dll 	DefaultFreeEntry 	nsprpub/lib/ds/plhash.c:87
4 	plds4.dll 	PL_HashTableDestroy 	nsprpub/lib/ds/plhash.c:147
5 	nssutil3.dll 	SECOID_Shutdown 	security/nss/lib/util/secoid.c:2138
6 	nss3.dll 	nss_Shutdown 	security/nss/lib/nss/nssinit.c:1033
7 	xul.dll 	nsNSSComponent::ShutdownNSS 	security/manager/ssl/src/nsNSSComponent.cpp:1925
8 	xul.dll 	nsNSSComponent::DoProfileBeforeChange 	security/manager/ssl/src/nsNSSComponent.cpp:2615
9 	xul.dll 	nsNSSComponent::Observe 	security/manager/ssl/src/nsNSSComponent.cpp:2244
10 	xul.dll 	nsObserverList::NotifyObservers 	xpcom/ds/nsObserverList.cpp:130
11 	xul.dll 	nsObserverService::NotifyObservers 	xpcom/ds/nsObserverService.cpp:182
12 	xul.dll 	xul.dll@0xb617cb 	
13 	xul.dll 	ScopedXPCOMStartup::~ScopedXPCOMStartup 	toolkit/xre/nsAppRunner.cpp:1073
14 	xul.dll 	XRE_main 	toolkit/xre/nsAppRunner.cpp:3722
15 	firefox.exe 	wmain 	toolkit/xre/nsWindowsWMain.cpp:106
16 	firefox.exe 	__tmainCRTStartup 	obj-firefox/memory/jemalloc/crtsrc/crtexe.c:591
17 	kernel32.dll 	BaseProcessStart 	

https://crash-stats.mozilla.com/report/list?signature=arena_dalloc_small%20%7C%20arena_dalloc%20%7C%20free%20%7C%20DefaultFreeEntry has more of those reports.

This has been rising significantly in the last days on Firefox 6, interestingly mostly matching adoption of 6.0b3 by beta users. It's not in the top 100 crashes for 6.0b2, but #34 for 6.0b3.

I haven't seen any NSS changes between those two releases, but the stack so much looks like it's NSS this comes from. Still, as I don't see NSS changesets between those two builds and want to request tracking for 6, filing in Core instead of NSS for now.
Summary: crash arena_dalloc_small → crash in arena_dalloc_small coming from ShutdownNSS
This looks like possible volume a regression, though the level is probably ok to ship with. We'll watch this, tracking6.

Can you pull URLs and try to reproduce? Adding qa keywords
OS: Linux → Windows 7
One comment says it happens in safe mode
So, up to today, this is #21 on 6.0b3, #18 in 6.0b4, #26 on 6.0b5.

(In reply to Christian Legnitto [:LegNeato] from comment #1)
> Can you pull URLs and try to reproduce? Adding qa keywords

I manually pulled up 20 recent crash reports and looked at their URLs, here they are:

1 http://www.youtube.com/js/pyv_watch_request_ad.html
6 http://www.facebook.com/logout.php
1 http://www.facebook.com/ai.php?aed=<foo>
1 http://twitter.com/?lang=en&logged_out=1#!/download
1 http://www.google.com.tw/search?q=<foo>
1 <URL omitted because it looks porn-ishlike>
1 https://support.netplus.fr/index.php
1 <no URL reported>
1 http://www.facebook.com/ajax/pagelet/generic.php/pagelet/home/morestories.php?<foo>
2 http://static.ak.fbcdn.net/connect/xd_proxy.php?<foo>
1 http://www.google.com.pk/imgres?q=<foo>
1 https://account.live.com/cobranding?id=<foo>
1 http://s7.addthis.com/static/<foo>.html#
1 http://<foo>.4shared.com/download/<foo>

<foo> is parts I stripped due to potential privacy risks.

To me, it looks like the URLs are just whatever was loaded when people closed the browser, esp. given the high numbers of the Facebook logout page, I start to assume this could be a shutdown crash.
Assignee: nobody → bsmith
Strangely I can't get Firefox to even hit facebook.com/logout.php. Clicking logout in the interface doesn't even touch this page, it goes directly to index.php.

Is there any way to reach out to people who are experiencing this crash to get more details (ie. profile data). I can't reproduce this at all locally.
Sitting at #20 on the 8.0 crash list for the past week and the #28 spot for 8.0.1.
Keywords: topcrash
(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #5)
> Strangely I can't get Firefox to even hit facebook.com/logout.php. Clicking
> logout in the interface doesn't even touch this page, it goes directly to
> index.php.
> 
> Is there any way to reach out to people who are experiencing this crash to
> get more details (ie. profile data). I can't reproduce this at all locally.

Perhaps disable javascript.
Duplicate of this bug: 711847
There have been no crashes in 9.0.1 and 10.0 Beta with the known signatures.
It's #27 top browser crasher in 11.0a2 and #70 in 12.0a1.
Crash Signature: [@ arena_dalloc_small | arena_dalloc | free | DefaultFreeEntry] → [@ arena_dalloc_small | arena_dalloc | free | DefaultFreeEntry] [@ je_free | DefaultFreeEntry]
It's #13 top browser crasher in 11.0b3 and #18 in 12.0a2.

The "arena_dalloc_small | arena_dalloc | free | DefaultFreeEntry" crash signature first appeared in 4.2a1pre/20110402. The regression range is:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=1a89509e25e4&tochange=b71e50bf9afc
It might be a regression from bug 640113 which is a video/audio bug.

It's correlated to media:
  je_free | DefaultFreeEntry|EXCEPTION_ACCESS_VIOLATION_READ (127 crashes)
     63% (80/127) vs.  23% (6857/29707) gkmedias.dll

Some comments talk about Flash Player.
Hardware: x86_64 → x86
Version: unspecified → 5 Branch
Duplicate of this bug: 706042
Summary: crash in arena_dalloc_small coming from ShutdownNSS → crash in DefaultFreeEntry coming from ShutdownNSS
Sitting at #11 on Fx12 for signature "je_free | DefaultFreeEntry". Looks like the "arena_dalloc_small | arena_dalloc | free | DefaultFreeEntry" disappeared after 8.0.1.
Crash Signature: [@ arena_dalloc_small | arena_dalloc | free | DefaultFreeEntry] [@ je_free | DefaultFreeEntry] → [@ arena_dalloc_small | arena_dalloc | free | DefaultFreeEntry] [@ je_free | DefaultFreeEntry] [@ arena_dalloc | __wrap_free | PR_Free | DefaultFreeEntry]
OS: Windows 7 → All
Hardware: x86 → All
Whiteboard: [native-crash]
Crash Signature: [@ arena_dalloc_small | arena_dalloc | free | DefaultFreeEntry] [@ je_free | DefaultFreeEntry] [@ arena_dalloc | __wrap_free | PR_Free | DefaultFreeEntry] → [@ arena_dalloc_small | arena_dalloc | free | DefaultFreeEntry] [@ je_free | DefaultFreeEntry] [@ arena_dalloc | __wrap_free | PR_Free | DefaultFreeEntry] [@ arena_dalloc_small | je_free | DefaultFreeEntry]
Crash Signature: [@ arena_dalloc_small | arena_dalloc | free | DefaultFreeEntry] [@ je_free | DefaultFreeEntry] [@ arena_dalloc | __wrap_free | PR_Free | DefaultFreeEntry] [@ arena_dalloc_small | je_free | DefaultFreeEntry] → [@ arena_dalloc_small | arena_dalloc | free | DefaultFreeEntry] [@ je_free | DefaultFreeEntry] [@ arena_dalloc | __wrap_free | PR_Free | DefaultFreeEntry] [@ arena_dalloc_small | je_free | DefaultFreeEntry ]
This appears to be hitting an assertion I added to jemalloc.  Note that the crash reason in [1] is EXCEPTION_BREAKPOINT, which means we hit a MOZ_CRASH.

If we could disassemble the code and pinpoint which assertion was being hit, that would give us insight into what's going wrong.

[1] https://crash-stats.mozilla.com/report/index/b9676525-3f94-49cd-848c-0ea612120620
bsmedberg, you disassembled one of the assertions before; would you mind looking at this one?  This crash is still happening; wasn't fixed by bug 766173.
One possibility is that this is a double-free resulting from calling NSS_Shutdown twice. I think there is a situation (where nsNSSComponent attempts to veto the profile change) where we may call NSS_Shutdown twice.
profile change veto has not been supported since Firefox 0.9.3, so if PSM is trying to veto profile change then it certainly might end up being confused like that.
Blocks: 718575
Crash Signature: [@ arena_dalloc_small | arena_dalloc | free | DefaultFreeEntry] [@ je_free | DefaultFreeEntry] [@ arena_dalloc | __wrap_free | PR_Free | DefaultFreeEntry] [@ arena_dalloc_small | je_free | DefaultFreeEntry ] → [@ arena_dalloc_small | arena_dalloc | free | DefaultFreeEntry] [@ je_free | DefaultFreeEntry] [@ arena_dalloc | __wrap_free | PR_Free | DefaultFreeEntry] [@ arena_dalloc_small | je_free | DefaultFreeEntry ] [@ arena_dalloc_small | arena_run_dalloc | …
It's #26 top browser crasher in 13.0.1 and 14.0b11, but jumps up to #10 in 15.0a2 and #14 in 16.0a1.

It started spiking in 15.0a1/20120524. The regression window for the spike is:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=36e938e51481&tochange=f43e8d300f21
It might be a regression from NSPR_4_9_1_BETA2.

The spike ended in 16.0a1/20120709. The working window is:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=1751d97cc9e4&tochange=b6fb3d9bd1b9

More reports at:
https://crash-stats.mozilla.com/report/list?signature=arena_dalloc_small+|+je_free+|+DefaultFreeEntry
(In reply to Scoobidiver from comment #17)
> It's #26 top browser crasher in 13.0.1 and 14.0b11, but jumps up to #10 in
> 15.0a2 and #14 in 16.0a1.
> 
> It started spiking in 15.0a1/20120524. The regression window for the spike
> is:
> http://hg.mozilla.org/mozilla-central/
> pushloghtml?fromchange=36e938e51481&tochange=f43e8d300f21
> It might be a regression from NSPR_4_9_1_BETA2.
> 
> The spike ended in 16.0a1/20120709. The working window is:
> http://hg.mozilla.org/mozilla-central/
> pushloghtml?fromchange=1751d97cc9e4&tochange=b6fb3d9bd1b9
> 
> More reports at:
> https://crash-stats.mozilla.com/report/
> list?signature=arena_dalloc_small+|+je_free+|+DefaultFreeEntry

Hey Scoobidiver - does the correlation remain to to gkmedias.dll as in https://bugzilla.mozilla.org/show_bug.cgi?id=675260#c10?
(In reply to Alex Keybl [:akeybl] from comment #18)
> Hey Scoobidiver - does the correlation remain to to gkmedias.dll as in
> https://bugzilla.mozilla.org/show_bug.cgi?id=675260#c10?
It's an internal Firefox DLL so I am not sure it's a relevant correlation. When I wrote comment 10, I'd hope it was correlated to HTML5 videos, but it seems loaded more often than this case. Anyway, here are correlations:
*13.0.1: 100% (563/563) vs.  58% (107305/185104) gkmedias.dll
*14.0 Beta: no correlation to gkmedias.dll
*15.0a2: 100% (58/58) vs.  92% (5007/5426) gkmedias.dll
*16.0a1: no correlation to gkmedias.dll
I took https://crash-stats.mozilla.com/report/index/d2f3329f-c1f7-43a1-8ca2-f8d7e2120716

From the disassembly, it appears that this assert is the one failing:

http://hg.mozilla.org/mozilla-central/annotate/57abb5f70e01/memory/mozjemalloc/jemalloc.c#l3308

If there's any other information you think I can glean from the minidump, let me know!
I think this is a double-free; it's the same assertion as bug 764192 comment 24.
Duplicate of this bug: 776050
Mozilla/5.0 (Windows NT 5.1; rv:15.0) Gecko/20100101 Firefox/15.0

Firefox 15 beta 2 crashed with the signature: Firefox 15.0 Crash Report [@ arena_dalloc_small | je_free | DefaultFreeEntry ] 
https://crash-stats.mozilla.com/report/index/bp-9389ecff-669d-4daa-a520-ea11b2120726

The crash occurred while closing the browser. In the moment of the crash I had 3 tabs loaded: one YouTube video, one with IMDb trailers and one Facebook game. 

I could not reproduce the crash afterwards.
(In reply to Justin Lebar [:jlebar] (PTO July 26) from comment #21)
> I think this is a double-free; it's the same assertion as bug 764192 comment
> 24.

jlebar - is there anything we can enable in Simona's browser in case she's able to repro in the future, to get better actionable info?
> jlebar - is there anything we can enable in Simona's browser in case she's able to repro in the 
> future, to get better actionable info?

Not unless Simona wants to browse in Valgrind until she hits this crash the next time, and that is a unique pain I would not wish on anyone.  :)

We have a reasonable guess as to what's causing this in comment 15; that's a place to start.
Crash Signature: je_free | DefaultFreeEntry ] → je_free | DefaultFreeEntry ] [@ arena_dalloc | libsystem_c.dylib@0x2d8c7]
Blocks: 780876
Brian - this is tracked for Firefox 15. Is there anything else we can do here for 15, or should we consider dropping the tracking flag?
There have been 151 reports of this crash in the last week, however all of them are on Firefox 8.0.1 or older. I can find no reports for a more recent version. Do we still need to track this for Firefox 15?
(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #27)
> There have been 151 reports of this crash in the last week
I count 12,000 crashes: https://crash-stats.mozilla.com/query/query?product=Firefox&query_search=signature&query_type=contains&query=DefaultFreeEntry&do_query=1
It's #8 top browser crasher in 15.0b4: https://crash-stats.mozilla.com/topcrasher/byversion/Firefox/15.0b4
Thanks Scoobidiver. Looking at that data I'm not seeing any useful correlations or comments that we can use to investigate. Do you have any suggestions?
(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #29)
> Do you have any suggestions?
71% of crashes occur on Windows XP.
Try to install the same extensions and modules of some people who crash on exit or on update.
(In reply to Scoobidiver from comment #30)
> 71% of crashes occur on Windows XP.
> Try to install the same extensions and modules of some people who crash on
> exit or on update.

Sure, I can try that. There was some developer follow-up that was supposed to happen in comment 15; any status update on that?
Can I get some help identifying the commonly installed extensions in the crash reports? It's kind of tedious to pull them out and cross reference them manually.
Crash Signature: je_free | DefaultFreeEntry ] [@ arena_dalloc | libsystem_c.dylib@0x2d8c7] → je_free | DefaultFreeEntry ] [@ arena_dalloc | libsystem_c.dylib@0x2d8c7] [@ libmozglue.so@0x8a2c] [@ libmozglue.so@0x898a] [@ libmozglue.so@0x8a80]
Crash Signature: [@ arena_dalloc_small | arena_dalloc | free | DefaultFreeEntry] [@ je_free | DefaultFreeEntry] [@ arena_dalloc | __wrap_free | PR_Free | DefaultFreeEntry] [@ arena_dalloc_small | je_free | DefaultFreeEntry ] [@ arena_dalloc_small | arena_run_dalloc | … → [@ arena_dalloc_small | arena_dalloc | free | DefaultFreeEntry] [@ je_free | DefaultFreeEntry] [@ moz_abort | je_free | DefaultFreeEntry] [@ arena_dalloc | __wrap_free | PR_Free | DefaultFreeEntry] [@ arena_dalloc_small | je_free | DefaultFreeEntry ] …
BTW: This crash just happened for me on Android, it crashed after I had quit the web app Galactians2 (https://marketplace.mozilla.org/app/galactians2/?src=). I was using the latest Firefox Aurora build for the game.
Spoke with Brian about this, it doesn't appear as if we have any actionable leads.
For future reference, there is no need to do this manually - https://crash-analysis.mozilla.com/crash_analysis/ has the manual correlations broken down by date (addon and module).

(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #32)
> Can I get some help identifying the commonly installed extensions in the
> crash reports? It's kind of tedious to pull them out and cross reference
> them manually.
This might be being caused by some code that does this:

nsNSSShutDownPreventionLock locker;
<Use NSS without checking whether NSS has already been shut down>

Though, I would expect the crash to happen in that other code, not in ShutdownNSS.
This is rising again on 16.0.1, possibly as release users switch over to it - doesn't seem to be higher than in 15, though.
Also affects 17 on Firefox for Android/Desktop
And it's #10 on Firefox desktop 17.0b1
Still sitting at #10 crash in FF 17 Beta 4 if you filter by just browser crashes.
https://crash-stats.mozilla.com/report/index/bp-2cb4cf6e-60bf-48dd-aa82-df3a42121115

Firefox 17 beta 6 crashed for me on Windows 7 64-bit with the above signature. The crash occured when launching Firefox after a Flash update.

STR:
1. Launched Firefox and pinned 4 tabs (Facebook, YouTube, Yahoo Mail and Google maps with the enabled WEB GL)
2. Opened 2 more tabs and loaded 2 Facebook games in each tab (Zuma Blitz and Cafe World)
3. Went to the Add-ons Manager -> Plug-ins -> checked for updates (Flash plug-in was out of date)
4. Downloaded the latest Flash version and installed it.
5. After the install was done I closed Firefox and launched it again.

Actual results:
At start up Firefox crashed.
1. Installed Flash 11.4.402.287
2. Installed Firefox 17.0b6 and started with a new profile
3. In the first tab, navigate to www.facebook.com, log in, and pin the tab
4. In the second tab, navigate to www.youtube.com, play a video, and pin the tab
5. In the third tab, navigate to mail.yahoo.com, log in, and pin the tab
6. In the forth tab, navigate to maps.google.com, enable MapsGL, and pin the tab
7. Switch back to the Facebook tab and launch two games in new tabs
8. Open Add-ons Manager > Plugins, click the link to check plugins are up to date
> PFS opens in a new tab and Flash is detected as "vulnerable"
9. Click "update now" button
> Redirected to Adobe's download page for Flash 11.5.502.110
10. Click the "download now" button and download Flash 11.5.502.110
11. Run the Flash 11.5.502.110 installer
> New tab opens to Adobe's installation successful page
12. Click the Firefox button > Exit
13. Double-click Firefox icon on desktop
> Firefox starts up and loads pinned tabs
14. Click "restore session" on the start page
> All other tabs are loaded

I've run through these steps a couple of different ways:
 * multiple times with new profiles
 * multiple times on the same profile
 * having the session restore completely on startup
 * having the session restore from the start page on startup

So far I've been unable to replicate Simona's results. Is this reproducible for you Simona?

Looking into the crash stats, does this mean this is trailing off?
 * #7 in 17.0b4
 * #9 in 17.0b5
 * #13 in 17.0b6

Can someone make a more astute evaluation of this crash in the wild? 
Is there anything else we can do for Firefox 17, given that RCs are building?
Keywords: steps-wanted
(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #42)
> Can someone make a more astute evaluation of this crash in the wild? 
> Is there anything else we can do for Firefox 17, given that RCs are building?

This has been a top crasher since FF15 - we will not be taking action in FF17. Engineering has not been able to figure out a next action here, given a lack of reproducible steps.
(In reply to Alex Keybl [:akeybl] from comment #43)
> This has been a top crasher since FF15 - we will not be taking action in
> FF17. Engineering has not been able to figure out a next action here, given
> a lack of reproducible steps.

Should the status flags be update to reflect this state?
(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #42)
> So far I've been unable to replicate Simona's results. Is this reproducible
> for you Simona?

I've tried to reproduce the crash afterwards but with no luck so far.
Given that we are still no closer to finding a reproducible case for this bug I think we should mark wontfix for Firefox 17 and start tracking for Firefox 18.
I don't think tracking will get us any closer to resolution, sadly.
Crash Signature: [@ arena_dalloc_small | arena_dalloc | free | DefaultFreeEntry] [@ je_free | DefaultFreeEntry] [@ moz_abort | je_free | DefaultFreeEntry] [@ arena_dalloc | __wrap_free | PR_Free | DefaultFreeEntry] [@ arena_dalloc_small | je_free | DefaultFreeEntry ] … → [@ arena_dalloc_small | arena_dalloc | free | DefaultFreeEntry] [@ arena_dalloc_small | arena_dalloc | je_free | DefaultFreeEntry] [@ je_free | DefaultFreeEntry] [@ moz_abort | je_free | DefaultFreeEntry] [@ arena_dalloc | __wrap_free | PR_Free | Defa…
Whiteboard: [native-crash] → [native-crash][tbird topcrash]
Crash Signature: DefaultFreeEntry] [@ arena_dalloc_small | je_free | DefaultFreeEntry ] [@ arena_dalloc_small | arena_run_dalloc | je_free | DefaultFreeEntry ] [@ arena_dalloc | libsystem_c.dylib@0x2d8c7] [@ libmozglue.so@0x8a2c] [@ libmozglue.so@0x898a] [@ libmozgl… → DefaultFreeEntry] [@ arena_dalloc_small | je_free | DefaultFreeEntry ] [@ arena_dalloc_small | arena_run_dalloc | je_free | DefaultFreeEntry ] [@ jemalloc_crash | arena_dalloc | __wrap_free | PR_Free | DefaultFreeEntry ] [@ arena_dalloc | libsystem_c.…
Brian, this keeps being pretty high, #8 topcrash on 18.0.1 and #5 on 19.0b3 at this time, is there anything that can be done here, instrumentation or whatever?
Flags: needinfo?(bsmith)
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #48)
> Brian, this keeps being pretty high, #8 topcrash on 18.0.1 and #5 on 19.0b3
> at this time, is there anything that can be done here, instrumentation or
> whatever?

rdow has taken the NSS shutdown crash bugs.
Assignee: bsmith → rdow
Flags: needinfo?(bsmith)
This is #17 on mobile 18.0.2 and #4 on desktop 18.0.2 right now. I guess bug 700499 is another related topcrash on mobile.

rdow, is there anything we can do to mitigate/reduce those (or at least find out where the culprit lies)?
Flags: needinfo?(rdow)
My opinion is that we have a data overwrite (probably not observing proper locking protocol) somewhere. I am working on a patch - which will only allow us to localize the problem at the moment.
Flags: needinfo?(rdow)
Crash Signature: libsystem_c.dylib@0x2d8c7] [@ libmozglue.so@0x8a2c] [@ libmozglue.so@0x898a] [@ libmozglue.so@0x8a80] → libsystem_c.dylib@0x2d8c7] [@ libmozglue.so@0x8a2c] [@ libmozglue.so@0x898a] [@ libmozglue.so@0x8a80] [@ jemalloc_crash | je_free | DefaultFreeEntry ]
(In reply to Rand Dow [:randix] from comment #51)
> My opinion is that we have a data overwrite (probably not observing proper
> locking protocol) somewhere. I am working on a patch - which will only allow
> us to localize the problem at the moment.

Did anything come out of that yet? This continues to be high in topcrash stats, e.g. #3 on 20.0b5 right now, #4 on 19.0.2 release.
That has yet to produce any real new insights directly. Indirectly, I am finding a lot of memory management cross-thread activity (I think). Tracking down the sources and verifying correct locking is proving difficult and time-consuming.
this is #6 on Release(Fx20), #3 on Beta(Fx21b), #12 on Aurora(Fx22a2), #23 and rising on Nightly(Fx23a1)

Looking at reports from the last week, Facebook and varios about: pages appear to be associated. I will dig around to see if I can uncover some sort of STR's.
Crash Signature: libsystem_c.dylib@0x2d8c7] [@ libmozglue.so@0x8a2c] [@ libmozglue.so@0x898a] [@ libmozglue.so@0x8a80] [@ jemalloc_crash | je_free | DefaultFreeEntry ] → libsystem_c.dylib@0x2d8c7] [@ libmozglue.so@0x8a2c] [@ libmozglue.so@0x898a] [@ libmozglue.so@0x8a80] [@ jemalloc_crash | je_free | DefaultFreeEntry ] [@ moz_abort | arena_run_reg_dalloc | arena_dalloc_small | arena_dalloc | je_free | DefaultFreeEnt…
(In reply to Rand Dow [:randix] from comment #54)
> That has yet to produce any real new insights directly. Indirectly, I am
> finding a lot of memory management cross-thread activity (I think). Tracking
> down the sources and verifying correct locking is proving difficult and
> time-consuming.

Can someone help you there? If so, what skill level is required?
Dougt, please assign. IMO all of these NSS shutdown bugs are related, probably simply different manifestations of the same root cause.
Assignee: rdow → doug.turner
This has shot to #4 on Fx21 topcrash list.
Assignee: doug.turner → mhamrick
Removing the native-crash whiteboard so that it shows up in the topcrash query for desktop Firefox.

It's #8 top browser crasher in 22.0.
Whiteboard: [native-crash][tbird topcrash] → [tbird topcrash]
Assignee: mhamrick → nobody
Following up in email, this was unassigned without comment.
Keywords: topcrashtopcrash-win
This has fallen far outside of topcrash ranks for now.
Keywords: topcrash-win
For Thunderbird, after version 24/starting with 31 there are no crash sigs containing DefaultFreeEntry.

For Firefox 36 and newer, the stack exists with signature [@ jemalloc_crash | arena_dalloc_small | je_free | DefaultFreeEntry ]
bp-dfe7c6f8-176d-484b-be88-3c91a2150129
0	mozglue.dll	jemalloc_crash	memory/mozjemalloc/jemalloc.c
1	mozglue.dll	arena_dalloc_small	memory/mozjemalloc/jemalloc.c
2	mozglue.dll	je_free	memory/mozjemalloc/jemalloc.c
3	nss3.dll	DefaultFreeEntry	nsprpub/lib/ds/plhash.c
4	nss3.dll	PL_HashTableDestroy	nsprpub/lib/ds/plhash.c
5	nss3.dll	nss3.dll@0x3aa9f	
6	nss3.dll	SECOID_Shutdown	security/nss/lib/util/secoid.c
7	nss3.dll	nss_Shutdown	security/nss/lib/nss/nssinit.c
8	nss3.dll	NSS_Shutdown	security/nss/lib/nss/nssinit.c
9	xul.dll	nsNSSComponent::ShutdownNSS()	security/manager/ssl/src/nsNSSComponent.cpp
10	xul.dll	nsNSSComponent::DoProfileBeforeChange(nsISupports*)	security/manager/ssl/src/nsNSSComponent.cpp

relatively rare, but 36 is still beta :)
https://crash-stats.mozilla.com/report/list?signature=jemalloc_crash+%7C+arena_dalloc_small+%7C+je_free+%7C+DefaultFreeEntry&product=Firefox&query_type=contains&range_unit=weeks&process_type=any&version=Firefox%3A38.0a1&version=Firefox%3A37.0a2&version=Firefox%3A36.0b&version=Firefox%3A35.0&version=Firefox%3A36.0b8&version=Firefox%3A36.0b7&version=Firefox%3A36.0b6&version=Firefox%3A36.0b5&version=Firefox%3A36.0b4&version=Firefox%3A36.0b3&version=Firefox%3A36.0b2&version=Firefox%3A36.0b1&version=Firefox%3A35.0.1&version=Firefox%3A34.0.5&version=Firefox%3A34.0&version=Firefox%3A33.1.1&version=Firefox%3A33.1&version=Firefox%3A33.0.3&version=Firefox%3A33.0.2&version=Firefox%3A33.0.1&hang_type=any&date=2015-02-12+04%3A00%3A00&range_value=2#tab-reports
Crash Signature: DefaultFreeEntry ] [@ jemalloc_crash | arena_dalloc_small | arena_dalloc | DefaultFreeEntry ] → DefaultFreeEntry ] [@ jemalloc_crash | arena_dalloc_small | arena_dalloc | DefaultFreeEntry ] [@ jemalloc_crash | arena_dalloc_small | je_free | DefaultFreeEntry ]
Whiteboard: [tbird topcrash]
You need to log in before you can comment on or make changes to this bug.