675260 - crash in DefaultFreeEntry coming from ShutdownNSS

Status: RESOLVED WORKSFORME

(Core :: Security: PSM, defect, P3)

Description

[Robert Kaiser]

This bug was filed from the Socorro interface and is 
report bp-75cd558a-aa19-44d9-8d85-839c22110729 .
============================================================= 

0 	mozcrt19.dll 	arena_dalloc_small 	obj-firefox/memory/jemalloc/crtsrc/jemalloc.c:4045
1 	mozcrt19.dll 	arena_dalloc 	obj-firefox/memory/jemalloc/crtsrc/jemalloc.c:4173
2 	mozcrt19.dll 	free 	obj-firefox/memory/jemalloc/crtsrc/jemalloc.c:6037
3 	plds4.dll 	DefaultFreeEntry 	nsprpub/lib/ds/plhash.c:87
4 	plds4.dll 	PL_HashTableDestroy 	nsprpub/lib/ds/plhash.c:147
5 	nssutil3.dll 	SECOID_Shutdown 	security/nss/lib/util/secoid.c:2138
6 	nss3.dll 	nss_Shutdown 	security/nss/lib/nss/nssinit.c:1033
7 	xul.dll 	nsNSSComponent::ShutdownNSS 	security/manager/ssl/src/nsNSSComponent.cpp:1925
8 	xul.dll 	nsNSSComponent::DoProfileBeforeChange 	security/manager/ssl/src/nsNSSComponent.cpp:2615
9 	xul.dll 	nsNSSComponent::Observe 	security/manager/ssl/src/nsNSSComponent.cpp:2244
10 	xul.dll 	nsObserverList::NotifyObservers 	xpcom/ds/nsObserverList.cpp:130
11 	xul.dll 	nsObserverService::NotifyObservers 	xpcom/ds/nsObserverService.cpp:182
12 	xul.dll 	xul.dll@0xb617cb 	
13 	xul.dll 	ScopedXPCOMStartup::~ScopedXPCOMStartup 	toolkit/xre/nsAppRunner.cpp:1073
14 	xul.dll 	XRE_main 	toolkit/xre/nsAppRunner.cpp:3722
15 	firefox.exe 	wmain 	toolkit/xre/nsWindowsWMain.cpp:106
16 	firefox.exe 	__tmainCRTStartup 	obj-firefox/memory/jemalloc/crtsrc/crtexe.c:591
17 	kernel32.dll 	BaseProcessStart 	

https://crash-stats.mozilla.com/report/list?signature=arena_dalloc_small%20%7C%20arena_dalloc%20%7C%20free%20%7C%20DefaultFreeEntry has more of those reports.

This has been rising significantly in the last days on Firefox 6, interestingly mostly matching adoption of 6.0b3 by beta users. It's not in the top 100 crashes for 6.0b2, but #34 for 6.0b3.

I haven't seen any NSS changes between those two releases, but the stack so much looks like it's NSS this comes from. Still, as I don't see NSS changesets between those two builds and want to request tracking for 6, filing in Core instead of NSS for now.

Comment 1

[LegNeato]

This looks like possible volume a regression, though the level is probably ok to ship with. We'll watch this, tracking6.

Can you pull URLs and try to reproduce? Adding qa keywords

Comment 2

[[:Cww]]

One comment says it happens in safe mode

Comment 3

[Robert Kaiser]

So, up to today, this is #21 on 6.0b3, #18 in 6.0b4, #26 on 6.0b5.

(In reply to Christian Legnitto [:LegNeato] from comment #1)
> Can you pull URLs and try to reproduce? Adding qa keywords

I manually pulled up 20 recent crash reports and looked at their URLs, here they are:

1 http://www.youtube.com/js/pyv_watch_request_ad.html
6 http://www.facebook.com/logout.php
1 http://www.facebook.com/ai.php?aed=<foo>
1 http://twitter.com/?lang=en&logged_out=1#!/download
1 http://www.google.com.tw/search?q=<foo>
1 <URL omitted because it looks porn-ishlike>
1 https://support.netplus.fr/index.php
1 <no URL reported>
1 http://www.facebook.com/ajax/pagelet/generic.php/pagelet/home/morestories.php?<foo>
2 http://static.ak.fbcdn.net/connect/xd_proxy.php?<foo>
1 http://www.google.com.pk/imgres?q=<foo>
1 https://account.live.com/cobranding?id=<foo>
1 http://s7.addthis.com/static/<foo>.html#
1 http://<foo>.4shared.com/download/<foo>

<foo> is parts I stripped due to potential privacy risks.

To me, it looks like the URLs are just whatever was loaded when people closed the browser, esp. given the high numbers of the Facebook logout page, I start to assume this could be a shutdown crash.

Comment 4

[Marcia Knous [:marcia]]

I was able to run a URL report and it looks as if Facebook is the top URL:

18 http://www.facebook.com/logout.php
      3 about:blank
      2 \N
      2 http://www.orkut.com/Logout?msg=0&hl=pt-BR
      2 http://www.google.com/
      2 https://www.facebook.com/login.php?login_attempt=1
http://www.worldlingo.com/SiLVb2DaMEWz7t8gRHURigWH8JLEaY2xD/translation?wl_url=http%3A%2F%2Fwww.foxlingo.com%2F%2Fhelp%2F&wl_srclang=en&wl_trglang=es
      1 http://www.videos-caseros.es/
      1 http://www.uol.com.br/
      1 http://www.tuenti.com/?m=login
      1 http://www.torrentino.com/torrents/17796/download
      1 http://www.panli.com/
      1 http://www.overseaseducation.co.in:2095/3rdparty/squirrelmail/src/signout.php
      1 http://www.orkut.com.br/Home
      1 http://www.onet.pl/

Comment 5

[u279076]

Strangely I can't get Firefox to even hit facebook.com/logout.php. Clicking logout in the interface doesn't even touch this page, it goes directly to index.php.

Is there any way to reach out to people who are experiencing this crash to get more details (ie. profile data). I can't reproduce this at all locally.

Comment 6

[Sheila Mooney]

Sitting at #20 on the 8.0 crash list for the past week and the #28 spot for 8.0.1.

Comment 7

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #5)
> Strangely I can't get Firefox to even hit facebook.com/logout.php. Clicking
> logout in the interface doesn't even touch this page, it goes directly to
> index.php.
> 
> Is there any way to reach out to people who are experiencing this crash to
> get more details (ie. profile data). I can't reproduce this at all locally.

Perhaps disable javascript.

Comment 9

[Scoobidiver (away)]

There have been no crashes in 9.0.1 and 10.0 Beta with the known signatures.
It's #27 top browser crasher in 11.0a2 and #70 in 12.0a1.

Comment 10

[Scoobidiver (away)]

It's #13 top browser crasher in 11.0b3 and #18 in 12.0a2.

The "arena_dalloc_small | arena_dalloc | free | DefaultFreeEntry" crash signature first appeared in 4.2a1pre/20110402. The regression range is:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=1a89509e25e4&tochange=b71e50bf9afc
It might be a regression from bug 640113 which is a video/audio bug.

It's correlated to media:
  je_free | DefaultFreeEntry|EXCEPTION_ACCESS_VIOLATION_READ (127 crashes)
     63% (80/127) vs.  23% (6857/29707) gkmedias.dll

Some comments talk about Flash Player.

Comment 12

[Sheila Mooney]

Sitting at #11 on Fx12 for signature "je_free | DefaultFreeEntry". Looks like the "arena_dalloc_small | arena_dalloc | free | DefaultFreeEntry" disappeared after 8.0.1.

Comment 13

[Justin Lebar (not reading bugmail)]

This appears to be hitting an assertion I added to jemalloc.  Note that the crash reason in [1] is EXCEPTION_BREAKPOINT, which means we hit a MOZ_CRASH.

If we could disassemble the code and pinpoint which assertion was being hit, that would give us insight into what's going wrong.

[1] https://crash-stats.mozilla.com/report/index/b9676525-3f94-49cd-848c-0ea612120620

Comment 14

[Justin Lebar (not reading bugmail)]

bsmedberg, you disassembled one of the assertions before; would you mind looking at this one?  This crash is still happening; wasn't fixed by bug 766173.

Comment 15

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

One possibility is that this is a double-free resulting from calling NSS_Shutdown twice. I think there is a situation (where nsNSSComponent attempts to veto the profile change) where we may call NSS_Shutdown twice.

Comment 16

[Benjamin Smedberg]

profile change veto has not been supported since Firefox 0.9.3, so if PSM is trying to veto profile change then it certainly might end up being confused like that.

Comment 17

[Scoobidiver (away)]

It's #26 top browser crasher in 13.0.1 and 14.0b11, but jumps up to #10 in 15.0a2 and #14 in 16.0a1.

It started spiking in 15.0a1/20120524. The regression window for the spike is:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=36e938e51481&tochange=f43e8d300f21
It might be a regression from NSPR_4_9_1_BETA2.

The spike ended in 16.0a1/20120709. The working window is:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=1751d97cc9e4&tochange=b6fb3d9bd1b9

More reports at:
https://crash-stats.mozilla.com/report/list?signature=arena_dalloc_small+|+je_free+|+DefaultFreeEntry

Comment 18

[Alex Keybl [:akeybl]]

(In reply to Scoobidiver from comment #17)
> It's #26 top browser crasher in 13.0.1 and 14.0b11, but jumps up to #10 in
> 15.0a2 and #14 in 16.0a1.
> 
> It started spiking in 15.0a1/20120524. The regression window for the spike
> is:
> http://hg.mozilla.org/mozilla-central/
> pushloghtml?fromchange=36e938e51481&tochange=f43e8d300f21
> It might be a regression from NSPR_4_9_1_BETA2.
> 
> The spike ended in 16.0a1/20120709. The working window is:
> http://hg.mozilla.org/mozilla-central/
> pushloghtml?fromchange=1751d97cc9e4&tochange=b6fb3d9bd1b9
> 
> More reports at:
> https://crash-stats.mozilla.com/report/
> list?signature=arena_dalloc_small+|+je_free+|+DefaultFreeEntry

Hey Scoobidiver - does the correlation remain to to gkmedias.dll as in https://bugzilla.mozilla.org/show_bug.cgi?id=675260#c10?

Comment 19

[Scoobidiver (away)]

(In reply to Alex Keybl [:akeybl] from comment #18)
> Hey Scoobidiver - does the correlation remain to to gkmedias.dll as in
> https://bugzilla.mozilla.org/show_bug.cgi?id=675260#c10?
It's an internal Firefox DLL so I am not sure it's a relevant correlation. When I wrote comment 10, I'd hope it was correlated to HTML5 videos, but it seems loaded more often than this case. Anyway, here are correlations:
*13.0.1: 100% (563/563) vs.  58% (107305/185104) gkmedias.dll
*14.0 Beta: no correlation to gkmedias.dll
*15.0a2: 100% (58/58) vs.  92% (5007/5426) gkmedias.dll
*16.0a1: no correlation to gkmedias.dll

Comment 20

[Benjamin Smedberg]

I took https://crash-stats.mozilla.com/report/index/d2f3329f-c1f7-43a1-8ca2-f8d7e2120716

From the disassembly, it appears that this assert is the one failing:

http://hg.mozilla.org/mozilla-central/annotate/57abb5f70e01/memory/mozjemalloc/jemalloc.c#l3308

If there's any other information you think I can glean from the minidump, let me know!

Comment 21

[Justin Lebar (not reading bugmail)]

I think this is a double-free; it's the same assertion as bug 764192 comment 24.

Comment 23

[Simona Badau, Desktop Test Engineering]

Mozilla/5.0 (Windows NT 5.1; rv:15.0) Gecko/20100101 Firefox/15.0

Firefox 15 beta 2 crashed with the signature: Firefox 15.0 Crash Report [@ arena_dalloc_small | je_free | DefaultFreeEntry ] 
https://crash-stats.mozilla.com/report/index/bp-9389ecff-669d-4daa-a520-ea11b2120726

The crash occurred while closing the browser. In the moment of the crash I had 3 tabs loaded: one YouTube video, one with IMDb trailers and one Facebook game. 

I could not reproduce the crash afterwards.

Comment 24

[Alex Keybl [:akeybl]]

(In reply to Justin Lebar [:jlebar] (PTO July 26) from comment #21)
> I think this is a double-free; it's the same assertion as bug 764192 comment
> 24.

jlebar - is there anything we can enable in Simona's browser in case she's able to repro in the future, to get better actionable info?

Comment 25

[Justin Lebar (not reading bugmail)]

> jlebar - is there anything we can enable in Simona's browser in case she's able to repro in the 
> future, to get better actionable info?

Not unless Simona wants to browse in Valgrind until she hits this crash the next time, and that is a unique pain I would not wish on anyone.  :)

We have a reasonable guess as to what's causing this in comment 15; that's a place to start.

Comment 26

[Josh Aas]

Brian - this is tracked for Firefox 15. Is there anything else we can do here for 15, or should we consider dropping the tracking flag?

Comment 27

[u279076]

There have been 151 reports of this crash in the last week, however all of them are on Firefox 8.0.1 or older. I can find no reports for a more recent version. Do we still need to track this for Firefox 15?

Comment 28

[Scoobidiver (away)]

(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #27)
> There have been 151 reports of this crash in the last week
I count 12,000 crashes: https://crash-stats.mozilla.com/query/query?product=Firefox&query_search=signature&query_type=contains&query=DefaultFreeEntry&do_query=1
It's #8 top browser crasher in 15.0b4: https://crash-stats.mozilla.com/topcrasher/byversion/Firefox/15.0b4

Comment 29

[u279076]

Thanks Scoobidiver. Looking at that data I'm not seeing any useful correlations or comments that we can use to investigate. Do you have any suggestions?

Comment 30

[Scoobidiver (away)]

(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #29)
> Do you have any suggestions?
71% of crashes occur on Windows XP.
Try to install the same extensions and modules of some people who crash on exit or on update.

Comment 31

[u279076]

(In reply to Scoobidiver from comment #30)
> 71% of crashes occur on Windows XP.
> Try to install the same extensions and modules of some people who crash on
> exit or on update.

Sure, I can try that. There was some developer follow-up that was supposed to happen in comment 15; any status update on that?

Comment 32

[u279076]

Can I get some help identifying the commonly installed extensions in the crash reports? It's kind of tedious to pull them out and cross reference them manually.

Comment 33

[Frank Wein [:mcsmurf]]

BTW: This crash just happened for me on Android, it crashed after I had quit the web app Galactians2 (https://marketplace.mozilla.org/app/galactians2/?src=). I was using the latest Firefox Aurora build for the game.

Comment 34

[Alex Keybl [:akeybl]]

Spoke with Brian about this, it doesn't appear as if we have any actionable leads.

Comment 35

[Marcia Knous [:marcia]]

For future reference, there is no need to do this manually - https://crash-analysis.mozilla.com/crash_analysis/ has the manual correlations broken down by date (addon and module).

(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #32)
> Can I get some help identifying the commonly installed extensions in the
> crash reports? It's kind of tedious to pull them out and cross reference
> them manually.

Comment 36

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

This might be being caused by some code that does this:

nsNSSShutDownPreventionLock locker;
<Use NSS without checking whether NSS has already been shut down>

Though, I would expect the crash to happen in that other code, not in ShutdownNSS.

Comment 37

[Robert Kaiser]

This is rising again on 16.0.1, possibly as release users switch over to it - doesn't seem to be higher than in 15, though.

Comment 38

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Also affects 17 on Firefox for Android/Desktop

Comment 39

[Robert Kaiser]

And it's #10 on Firefox desktop 17.0b1

Comment 40

[Marcia Knous [:marcia]]

Still sitting at #10 crash in FF 17 Beta 4 if you filter by just browser crashes.

Comment 41

[Simona Badau, Desktop Test Engineering]

https://crash-stats.mozilla.com/report/index/bp-2cb4cf6e-60bf-48dd-aa82-df3a42121115

Firefox 17 beta 6 crashed for me on Windows 7 64-bit with the above signature. The crash occured when launching Firefox after a Flash update.

STR:
1. Launched Firefox and pinned 4 tabs (Facebook, YouTube, Yahoo Mail and Google maps with the enabled WEB GL)
2. Opened 2 more tabs and loaded 2 Facebook games in each tab (Zuma Blitz and Cafe World)
3. Went to the Add-ons Manager -> Plug-ins -> checked for updates (Flash plug-in was out of date)
4. Downloaded the latest Flash version and installed it.
5. After the install was done I closed Firefox and launched it again.

Actual results:
At start up Firefox crashed.

Comment 42

[u279076]

1. Installed Flash 11.4.402.287
2. Installed Firefox 17.0b6 and started with a new profile
3. In the first tab, navigate to www.facebook.com, log in, and pin the tab
4. In the second tab, navigate to www.youtube.com, play a video, and pin the tab
5. In the third tab, navigate to mail.yahoo.com, log in, and pin the tab
6. In the forth tab, navigate to maps.google.com, enable MapsGL, and pin the tab
7. Switch back to the Facebook tab and launch two games in new tabs
8. Open Add-ons Manager > Plugins, click the link to check plugins are up to date
> PFS opens in a new tab and Flash is detected as "vulnerable"
9. Click "update now" button
> Redirected to Adobe's download page for Flash 11.5.502.110
10. Click the "download now" button and download Flash 11.5.502.110
11. Run the Flash 11.5.502.110 installer
> New tab opens to Adobe's installation successful page
12. Click the Firefox button > Exit
13. Double-click Firefox icon on desktop
> Firefox starts up and loads pinned tabs
14. Click "restore session" on the start page
> All other tabs are loaded

I've run through these steps a couple of different ways:
 * multiple times with new profiles
 * multiple times on the same profile
 * having the session restore completely on startup
 * having the session restore from the start page on startup

So far I've been unable to replicate Simona's results. Is this reproducible for you Simona?

Looking into the crash stats, does this mean this is trailing off?
 * #7 in 17.0b4
 * #9 in 17.0b5
 * #13 in 17.0b6

Can someone make a more astute evaluation of this crash in the wild? 
Is there anything else we can do for Firefox 17, given that RCs are building?

Comment 43

[Alex Keybl [:akeybl]]

(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #42)
> Can someone make a more astute evaluation of this crash in the wild? 
> Is there anything else we can do for Firefox 17, given that RCs are building?

This has been a top crasher since FF15 - we will not be taking action in FF17. Engineering has not been able to figure out a next action here, given a lack of reproducible steps.

Comment 44

[u279076]

(In reply to Alex Keybl [:akeybl] from comment #43)
> This has been a top crasher since FF15 - we will not be taking action in
> FF17. Engineering has not been able to figure out a next action here, given
> a lack of reproducible steps.

Should the status flags be update to reflect this state?

Comment 45

[Simona Badau, Desktop Test Engineering]

(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #42)
> So far I've been unable to replicate Simona's results. Is this reproducible
> for you Simona?

I've tried to reproduce the crash afterwards but with no luck so far.

Comment 46

[u279076]

Given that we are still no closer to finding a reproducible case for this bug I think we should mark wontfix for Firefox 17 and start tracking for Firefox 18.

Comment 47

[Alex Keybl [:akeybl]]

I don't think tracking will get us any closer to resolution, sadly.

Comment 48

[Robert Kaiser]

Brian, this keeps being pretty high, #8 topcrash on 18.0.1 and #5 on 19.0b3 at this time, is there anything that can be done here, instrumentation or whatever?

Comment 49

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Robert Kaiser (:[e-mail]) from comment #48)
> Brian, this keeps being pretty high, #8 topcrash on 18.0.1 and #5 on 19.0b3
> at this time, is there anything that can be done here, instrumentation or
> whatever?

rdow has taken the NSS shutdown crash bugs.

Comment 50

[Robert Kaiser]

This is #17 on mobile 18.0.2 and #4 on desktop 18.0.2 right now. I guess bug 700499 is another related topcrash on mobile.

rdow, is there anything we can do to mitigate/reduce those (or at least find out where the culprit lies)?

Comment 51

[Rand Dow [:randix]]

My opinion is that we have a data overwrite (probably not observing proper locking protocol) somewhere. I am working on a patch - which will only allow us to localize the problem at the moment.

Comment 52

[Rand Dow [:randix]]

https://bugzilla.mozilla.org/show_bug.cgi?id=700499#c29

Comment 53

[Robert Kaiser]

(In reply to Rand Dow [:randix] from comment #51)
> My opinion is that we have a data overwrite (probably not observing proper
> locking protocol) somewhere. I am working on a patch - which will only allow
> us to localize the problem at the moment.

Did anything come out of that yet? This continues to be high in topcrash stats, e.g. #3 on 20.0b5 right now, #4 on 19.0.2 release.

Comment 54

[Rand Dow [:randix]]

That has yet to produce any real new insights directly. Indirectly, I am finding a lot of memory management cross-thread activity (I think). Tracking down the sources and verifying correct locking is proving difficult and time-consuming.

Comment 55

[Tracy Walker [:tracy]]

this is #6 on Release(Fx20), #3 on Beta(Fx21b), #12 on Aurora(Fx22a2), #23 and rising on Nightly(Fx23a1)

Looking at reports from the last week, Facebook and varios about: pages appear to be associated. I will dig around to see if I can uncover some sort of STR's.

Comment 56

[Robert Kaiser]

(In reply to Rand Dow [:randix] from comment #54)
> That has yet to produce any real new insights directly. Indirectly, I am
> finding a lot of memory management cross-thread activity (I think). Tracking
> down the sources and verifying correct locking is proving difficult and
> time-consuming.

Can someone help you there? If so, what skill level is required?

Comment 57

[Rand Dow [:randix]]

Dougt, please assign. IMO all of these NSS shutdown bugs are related, probably simply different manifestations of the same root cause.

Comment 58

[Tracy Walker [:tracy]]

This has shot to #4 on Fx21 topcrash list.

Comment 59

[Scoobidiver (away)]

Removing the native-crash whiteboard so that it shows up in the topcrash query for desktop Firefox.

It's #8 top browser crasher in 22.0.

Comment 60

[Alex Keybl [:akeybl]]

Following up in email, this was unassigned without comment.

Comment 61

[Robert Kaiser]

This has fallen far outside of topcrash ranks for now.

Comment 62

[Wayne Mery (:wsmwk)]

For Thunderbird, after version 24/starting with 31 there are no crash sigs containing DefaultFreeEntry.

For Firefox 36 and newer, the stack exists with signature [@ jemalloc_crash | arena_dalloc_small | je_free | DefaultFreeEntry ]
bp-dfe7c6f8-176d-484b-be88-3c91a2150129
0	mozglue.dll	jemalloc_crash	memory/mozjemalloc/jemalloc.c
1	mozglue.dll	arena_dalloc_small	memory/mozjemalloc/jemalloc.c
2	mozglue.dll	je_free	memory/mozjemalloc/jemalloc.c
3	nss3.dll	DefaultFreeEntry	nsprpub/lib/ds/plhash.c
4	nss3.dll	PL_HashTableDestroy	nsprpub/lib/ds/plhash.c
5	nss3.dll	nss3.dll@0x3aa9f	
6	nss3.dll	SECOID_Shutdown	security/nss/lib/util/secoid.c
7	nss3.dll	nss_Shutdown	security/nss/lib/nss/nssinit.c
8	nss3.dll	NSS_Shutdown	security/nss/lib/nss/nssinit.c
9	xul.dll	nsNSSComponent::ShutdownNSS()	security/manager/ssl/src/nsNSSComponent.cpp
10	xul.dll	nsNSSComponent::DoProfileBeforeChange(nsISupports*)	security/manager/ssl/src/nsNSSComponent.cpp

relatively rare, but 36 is still beta :)
https://crash-stats.mozilla.com/report/list?signature=jemalloc_crash+%7C+arena_dalloc_small+%7C+je_free+%7C+DefaultFreeEntry&product=Firefox&query_type=contains&range_unit=weeks&process_type=any&version=Firefox%3A38.0a1&version=Firefox%3A37.0a2&version=Firefox%3A36.0b&version=Firefox%3A35.0&version=Firefox%3A36.0b8&version=Firefox%3A36.0b7&version=Firefox%3A36.0b6&version=Firefox%3A36.0b5&version=Firefox%3A36.0b4&version=Firefox%3A36.0b3&version=Firefox%3A36.0b2&version=Firefox%3A36.0b1&version=Firefox%3A35.0.1&version=Firefox%3A34.0.5&version=Firefox%3A34.0&version=Firefox%3A33.1.1&version=Firefox%3A33.1&version=Firefox%3A33.0.3&version=Firefox%3A33.0.2&version=Firefox%3A33.0.1&hang_type=any&date=2015-02-12+04%3A00%3A00&range_value=2#tab-reports

Comment 64

[BugBot [:suhaib / :marco]]

Closing because no crashes reported for 12 weeks.