709860 - OOM crash in nsIDNService::Init @ mozilla::Preferences::GetBranch

Status: RESOLVED WORKSFORME

(Core :: Networking: DNS, defect)

Description

[Marcia Knous [:marcia]]

This bug was filed from the Socorro interface and is 
report bp-69c1b2ed-cbb4-4cd8-85fd-d3b782111210 .
============================================================= 

This ranks as the #33 crash in Firefox 9b5 data but did not have a bug associated with it. https://crash-stats.mozilla.com/report/list?signature=mozalloc_abort%28char%20const*%20const%29%20|%20mozalloc_handle_oom%28%29%20|%20mozilla::Preferences::GetBranch%28char%20const*,%20nsIPrefBranch**%29

Happened in small volume in 8.0 but has increased in Firefox 9 and happens up to b5. Will look for some correlations.

Frame 	Module 	Signature [Expand] 	Source
0 	mozalloc.dll 	mozalloc_abort 	memory/mozalloc/mozalloc_abort.cpp:77
1 	mozalloc.dll 	mozalloc_handle_oom 	memory/mozalloc/mozalloc_oom.cpp:54
2 	xul.dll 	mozilla::Preferences::GetBranch 	modules/libpref/src/Preferences.cpp:563
3 	xul.dll 	nsIDNService::Init 	netwerk/dns/nsIDNService.cpp:88
4 	xul.dll 	nsIDNServiceConstructor 	netwerk/build/nsNetModule.cpp:352
5 	xul.dll 	mozilla::GenericFactory::CreateInstance 	obj-firefox/xpcom/build/GenericFactory.cpp:48
6 	xul.dll 	nsComponentManagerImpl::CreateInstanceByContractID 	xpcom/components/nsComponentManager.cpp:1299
7 	xul.dll 	nsComponentManagerImpl::GetServiceByContractID 	xpcom/components/nsComponentManager.cpp:1701
8 	xul.dll 	nsCOMPtr_base::assign_from_gs_contractid 	obj-firefox/xpcom/build/nsCOMPtr.cpp:132
9 	xul.dll 	nsDNSService::Init 	netwerk/dns/nsDNSService2.cpp:431
10 	xul.dll 	nsDNSServiceConstructor 	netwerk/build/nsNetModule.cpp:85
11 	xul.dll 	mozilla::GenericFactory::CreateInstance 	obj-firefox/xpcom/build/GenericFactory.cpp:48
12 	xul.dll 	nsComponentManagerImpl::CreateInstanceByContractID 	xpcom/components/nsComponentManager.cpp:1299
13 	xul.dll 	nsComponentManagerImpl::GetServiceByContractID 	xpcom/components/nsComponentManager.cpp:1701
14 	xul.dll 	nsCOMPtr_base::assign_from_gs_contractid_with_error 	obj-firefox/xpcom/build/nsCOMPtr.cpp:141
15 	xul.dll 	nsIOService::Init 	netwerk/base/src/nsIOService.cpp:199
16 	xul.dll 	nsIOService::GetInstance 	netwerk/base/src/nsIOService.cpp:321
17 	xul.dll 	nsIOServiceConstructor 	netwerk/build/nsNetModule.cpp:82
18 	xul.dll 	mozilla::GenericFactory::CreateInstance 	obj-firefox/xpcom/build/GenericFactory.cpp:48
19 	xul.dll 	nsComponentManagerImpl::CreateInstanceByContractID 	xpcom/components/nsComponentManager.cpp:1299
20 	xul.dll 	nsComponentManagerImpl::GetServiceByContractID 	xpcom/components/nsComponentManager.cpp:1701
21 	xul.dll 	nsCOMPtr_base::assign_from_gs_contractid 	obj-firefox/xpcom/build/nsCOMPtr.cpp:132
22 	xul.dll 	mozilla::services::GetIOService 	xpcom/build/ServiceList.h:8
23 	xul.dll 	nsChromeRegistryChrome::ManifestResource 	chrome/src/nsChromeRegistryChrome.cpp:1044
24 	xul.dll 	ParseManifestCommon 	xpcom/components/ManifestParser.cpp:649
25 	xul.dll 	ParseManifest 	xpcom/components/ManifestParser.cpp:687
26 	xul.dll 	nsComponentManagerImpl::RegisterJarManifest 	xpcom/components/nsComponentManager.cpp:583
27 	xul.dll 	XRE_AddJarManifestLocation 	xpcom/components/nsComponentManager.cpp:2131
28 	xul.dll 	LoadExtensionDirectories 	
29 	xul.dll 	nsXREDirProvider::LoadExtensionBundleDirectories 	toolkit/xre/nsXREDirProvider.cpp:557
30 	xul.dll 	nsXREDirProvider::DoStartup 	toolkit/xre/nsXREDirProvider.cpp:734
31 	xul.dll 	XRE_main 	toolkit/xre/nsAppRunner.cpp:3429
32 	firefox.exe 	wmain 	toolkit/xre/nsWindowsWMain.cpp:107
33 	firefox.exe 	firefox.exe@0x4033 	
34 	firefox.exe 	__tmainCRTStartup 	crtexe.c:594
35 	firefox.exe 	_SEH_epilog4 	
36 	kernel32.dll 	kernel32.dll@0x51113 	
37 	ntdll.dll 	__RtlUserThreadStart 	
38 	kernel32.dll 	kernel32.dll@0x62acc 	
39 	kernel32.dll 	kernel32.dll@0x62acc 	
40 	ntdll.dll 	LdrpGetShimEngineInterface 	
41 	ntdll.dll 	_RtlUserThreadStart 	
42 	firefox.exe 	pre_c_init 	crtexe.c:304
43 	firefox.exe 	pre_c_init 	crtexe.c:304
44 		@0x7ffd5fff

Comment 1

[Scoobidiver (away)]

It's #121 top crasher in 9.0.1, #14 in 11.0a2, and #31 in 12.0a1.

Comment 2

[Muhammed Hasan]

A user's computer crashed and since then Firefox opens with Crash Reporter ( https://support.mozilla.org/en-US/questions/918384 )

Crash Report -> https://crash-stats.mozilla.com/report/index/9bc3a57b-aa15-4581-80b3-4bb122120210

Comment 3

[Scoobidiver (away)]

It's now #3 top browser crasher in 9.0.1 and #25 in 11.0b2.

Comment 4

[Benjamin Smedberg]

Does this have the same regression range as bug 725280?

Comment 5

[Scoobidiver (away)]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #4)
> Does this have the same regression range as bug 725280?
It's the same bug indeed. It's only the signature that has changed since 12.0.

Comment 7

[Benjamin Smedberg]

Do we have a nightly regression range for this signature?

Comment 8

[Josh Aas]

Marking this as tracking FF12.

Comment 9

[Scoobidiver (away)]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #7)
> Do we have a nightly regression range for this signature?
It was a residual signature in release versions but it started in Nightly from 9.0a1/20110923. The regression range might be:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=4495e1f795c2&tochange=259d1556c221

Comment 10

[Benjamin Smedberg]

That's a much more useful regression range than bug 725280! The two reasonable candidates here are:

the *backout* of bug 477578 (originally landed 2011-09-13, backed out 2011-09-22)
Bug 687722 - Make swapping two nsAutoTArrays preserve their auto-ness when possible

khuey, I know this was long ago, but do you remember whether bug 477578 was a straight backout or whether there were any fuzz/merge issues?

I don't see any autoarray swapping in the code leading up to this crash, but since we're probably looking at a memory corruption bug with the jemalloc 72-byte sizeclass, this could be a bug in code which runs well before the current stack trace. It's also possible that there is some thread race, although I haven't found a crash report with any other interesting threads running, and I don't think we've even started the I/O thread or the DNS threadpool when this crash occurs.

Comment 11

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #10)
> khuey, I know this was long ago, but do you remember whether bug 477578 was
> a straight backout or whether there were any fuzz/merge issues?

I don't recall.  Sorry.

Comment 12

[Benjamin Smedberg]

Given that this is not a new regression in Firefox 12, do you still want to track on it? I think there's something interesting going on, but unless this is really one of the high topcrashers I'm not sure it's worth tracking specifically on.

Comment 13

[Sheila Mooney]

bsmedberg, no I don't see a reason to track it for 12 so I will remove the flag. Looking at the signature summary report for the last week. Most of these are on 9.0.1. It's not really in any significant volume for 10.0.2. I will leave it as a top crash for now.

Comment 14

[Robert Kaiser]

This has been rising to #10 on Firefox 11.* in yesterday's data.

Comment 15

[Scoobidiver (away)]

It's #158 top crasher in 12.0, #12 in 13.0b6, #17 in 14.0a2.

Comment 16

[Alex Keybl [:akeybl]]

(In reply to Scoobidiver from comment #15)
> It's #158 top crasher in 12.0, #12 in 13.0b6, #17 in 14.0a2.

The most recent spike does appear new to FF13, but not to 13.0b6 specifically. We need to get URLs and a correlation report asap.

Comment 17

[Marcia Knous [:marcia]]

Several of the signatures such as mozalloc_abort(char const* const) | mozalloc_handle_oom(unsigned int) | moz_xmalloc | mozilla::Preferences::GetBranch(char const*, nsIPrefBranch**) show no URLs. The uptime is nonexistent so I guess it is not able to grab any kind of URL before the crash happens. Our best next step is to look at module correlations and comments, which I will do next.

Comment 18

[Marcia Knous [:marcia]]

Some module correlations for the first signature. I will attach the longer detailed version breakdown in an attachment.

mozalloc_abort(char const* const) | mozalloc_handle_oom(unsigned int) | moz_xmalloc | mozilla::Preferences::GetBranch(char const*, nsIPrefBranch**)|EXCEPTION_BREAKPOINT (335 crashes)
     99% (333/335) vs.  73% (32514/44497) browsercomps.dll
    100% (335/335) vs.  74% (33037/44497) firefox.exe
    100% (335/335) vs.  74% (33074/44497) xpcom.dll
    100% (335/335) vs.  75% (33337/44497) dbghelp.dll
     99% (330/335) vs.  74% (33033/44497) softokn3.dll
     32% (106/335) vs.  16% (6984/44497) snxhk.dll
     26% (88/335) vs.  14% (6341/44497) aswJsFlt.dll
     63% (212/335) vs.  54% (24148/44497) comres.dll
     63% (212/335) vs.  55% (24570/44497) ws2help.dll
     63% (212/335) vs.  55% (24573/44497) iphlpapi.dll
      7% (24/335) vs.   0% (114/44497) zvexescn.dll
      7% (24/335) vs.   0% (115/44497) ZVFORT.DLL

Comment 19

[Marcia Knous [:marcia]]

Attachment: Breakdown by module version

Comment 20

[Marcia Knous [:marcia]]

Some percentage of the users have Avast installed: 26% (88/335) vs.  14% (6341/44497) aswJsFlt.dll. The problem version seems to be specifically  
21% (70/335) vs.  12% (5284/44497) 7.0.1426.0

Comment 21

[Marcia Knous [:marcia]]

I installed Avast 7.0.1426.0 down on a Win XP lab machine, but so far have not been able to generate a crash yet.

Comment 22

[Lukas Blakk [:lsblakk] use ?needinfo]

This signature seems to have taken a nose-dive since 5/30 and is no longer on the topcrash list for 14.0a2 - can someone take a look and see if something was landed which would possibly have addressed this crash?

Comment 23

[Scoobidiver (away)]

With combined signatures, it's #5 top browser crasher in 13.0b7 and #9 in 14.0a2 over the last 3 days.

Comment 24

[Alex Keybl [:akeybl]]

I know Marcia is already looking into this, but adding qawanted for accuracy.

Comment 25

[Marcia Knous [:marcia]]

http://tinyurl.com/cy53rak is the signature in FF 13 Beta 7 that seems to have the most crashes, and it is a still a startup crash.

Still seems to be correlated to Avast:

mozalloc_abort(char const* const) | mozalloc_handle_oom(unsigned int) | moz_xmalloc | mozilla::Preferences::GetBranch(char const*, nsIPrefBranch**)|EXCEPTION_BREAKPOINT (340 crashes)
    100% (340/340) vs.  70% (21303/30293) firefox.exe
    100% (340/340) vs.  70% (21312/30293) xpcom.dll
    100% (340/340) vs.  71% (21532/30293) dbghelp.dll
     89% (304/340) vs.  69% (20929/30293) browsercomps.dll
     36% (123/340) vs.  17% (5032/30293) snxhk.dll
     89% (303/340) vs.  70% (21263/30293) softokn3.dll
     34% (115/340) vs.  15% (4563/30293) aswJsFlt.dll
     39% (133/340) vs.  29% (8821/30293) DWrite.dll
      9% (29/340) vs.   2% (512/30293) protector.dll
      6% (22/340) vs.   1% (370/30293) asoehook.dll

Comment 26

[Marcia Knous [:marcia]]

I have tried turning on all different Avast settings, running in Sandboxed mode etc and still haven't generated a crash. Because it is a startup crash there are no URLs to go from so that makes it a bit difficult.

Comment 27

[Jason Duell]

So my understanding is that this is a likely memory corruption bug causing malloc to barf in necko, but not necessarily a necko bug (if the regression range in comment 9 is correct, it's very unlikely to be in necko).   Is it time to ping the authors/reviewers of the bugs in the regression range and ask them to look over their commits for possible pointer errors?  I'm not sure how we handle these sorts of bugs, but I can't think of a better strategy in the absence of steps to repro.

Comment 28

[Robert Kaiser]

We don't have URLs as this is very early in startup when we don't have even the browser UI loaded yet.

Comment 29

[Marcia Knous [:marcia]]

For 13, the specific signature that showed up in the betas and 13 final is: mozalloc_abort(char const* const) | mozalloc_handle_oom(unsigned int) | moz_xmalloc | mozilla::Preferences::GetBranch(char const*, nsIPrefBranch**). http://tinyurl.com/7a6v6qo is the link to those crashes.

From what I can gather from the signature summary data, this signature has dropped off in volume a bit since Beta 7. Currently a 13 specific query is showing 300 crashes.

Comment 30

[Benjamin Smedberg]

jlebar, I'm pretty certain now that this is a memory corruption bug specific to the 72-byte jemalloc size class. I'm wondering if there is release-mode checking we can add *just* for this sizeclass that we could get into Fx14 to get a better handle on this? For example sanity-checking free() calls more aggressively, or memory-poisoning the data just in this sizeclass to detect the location of the double-free or read/write-after-free? If you're not the jemalloc expert to ask, please let me know who is ;-)

Comment 31

[Marcia Knous [:marcia]]

FF 13 query shows 894 crashes in the last week in this stack in mozalloc_abort(char const* const) | mozalloc_handle_oom(unsigned int) | moz_xmalloc | mozilla::Preferences::GetBranch(char const*, nsIPrefBranch**) signature.

Comment 32

[Justin Lebar (not reading bugmail)]

> For example sanity-checking free() calls more aggressively, or memory-poisoning the data just in 
> this sizeclass to detect the location of the double-free or read/write-after-free?

I think we could probably do that.  I'll get back to you in a few hours, once I have a look through the code.

Comment 33

[Justin Lebar (not reading bugmail)]

I've landed a patch on m-i to add these assertions.  I can land on aurora/beta once I get approval.

No guarantees that it'll work, but it's worth a shot...

Comment 34

[Alex Keybl [:akeybl]]

I know there's nothing actionable right now, but it appears you're heading up the engineering investigation Benjamin (hopefully bug 764192 helps). Sending over to you.

Comment 35

[Andrew McCreight [:mccr8]]

This sounds a little like the native Fennec crashes in bug 759674, bug 759675 and bug 759680.  Those crashes look to be caused by smallish mallocs (500 or 2000 bytes) failing.  Of course, it is mobile, so maybe they are legitimately failing, but it is odd we're seeing a ton of crashes right in XPCWrappedNativeScope creation, but not elsewhere.

Comment 36

[Virgil Dicu [:virgil] [QA]]

Mozilla/5.0 (Windows NT 6.1; rv:13.0) Gecko/20100101 Firefox/13.0

Also tried to reproduce this on Avast anti virus installed (Free-30 days protection). Unsuccessfull so far. Changed some settings related to browser protection, blocked some URLs, played a bit with the third party add-on installed with Avast and upgraded from 12 to 13. No crash.

Comment 37

[Justin Lebar (not reading bugmail)]

This may be fixed by bug 766173.

Comment 38

[Marcia Knous [:marcia]]

Alex asked me to follow up with rank - this crash still appears in 13.0.1 crash data in the #31 spot with 1248 crashes. Now that Bug 766173 landed we should keep an eye on Aurora data.

Comment 39

[Benjamin Smedberg]

I believe this signature morphed into bug 718575 on aurora and possibly trunk.

Comment 40

[Virgil Dicu [:virgil] [QA]]

Is there still any need for the qawanted keyword here? Any new information which we can use to reproduce? I could not reproduce in comment 36 with the Avast antivirus installed.

Comment 41

[Virgil Dicu [:virgil] [QA]]

Removing the qawanted keyword for the moment, as there is no new information based on which we could reproduce.

Comment 42

[Alex Keybl [:akeybl]]

We shipped FF13 with this crash, and sadly do not have more actionable data at this time. Wontfixing for FF14.

Comment 43

[Scoobidiver (away)]

There are no crashes in 15.0.1 and above.