508292 - [WinXP] crash [@ wcslen | HostentBlob_WriteNameOrAlias] under PR_GetHostByName

Status: RESOLVED INCOMPLETE

(Core :: Networking, defect, P2)

Description

[Martijn Wargers (dead)]

Currently nr. 2 in the topcrasher list for Firefox3.5.2:
http://crash-stats.mozilla.com/query/query?product=Firefox&version=Firefox%3A3.5.2&date=&range_value=1&range_unit=weeks&query_search=signature&query_type=exact&query=&do_query=1

This is a list of the stacks:
http://crash-stats.mozilla.com/report/list?product=Firefox&version=Firefox%3A3.5.2&query_search=signature&query_type=exact&query=&date=&range_value=1&range_unit=weeks&do_query=1&signature=PR_GetHostByName

Example stack:
http://crash-stats.mozilla.com/report/index/0984e159-39b7-4515-ad9f-677f42090804
0  	 	@0x5f90005  	
1 	nspr4.dll 	PR_GetHostByName 	nsprpub/pr/src/misc/prnetdb.c:722
2 	nspr4.dll 	pr_GetAddrInfoByNameFB 	nsprpub/pr/src/misc/prnetdb.c:1987
3 	nspr4.dll 	PR_GetAddrInfoByName 	nsprpub/pr/src/misc/prnetdb.c:2016
4 	xul.dll 	nsHostResolver::ThreadFunc 	netwerk/dns/src/nsHostResolver.cpp:884
5 	nspr4.dll 	_PR_NativeRunThread 	nsprpub/pr/src/threads/combined/pruthr.c:426
6 	xul.dll 	nsACString_internal::MutatePrep 	xpcom/string/src/nsTSubstring.cpp:179
7 	mozcrt19.dll 	_callthreadstartex 	obj-firefox/memory/jemalloc/src/threadex.c:348

The crash seems to happen at start up. Most cases the uptime is less than 10s.

I'm not sure if these crashes are the same, I think they are, they seem to give a better stack:
http://crash-stats.mozilla.com/report/list?product=Firefox&version=Firefox%3A3.5.2&query_search=signature&query_type=exact&query=&date=&range_value=1&range_unit=weeks&do_query=1&signature=wcslen

Example stack of these crashes:
http://crash-stats.mozilla.com/report/index/51eb5f20-c37e-4fa8-b831-fa2e12090804
0  	msvcrt.dll  	wcslen  	
1 	mswsock.dll 	mswsock.dll@0x9629 	
2 	mswsock.dll 	mswsock.dll@0x9809 	
3 	mswsock.dll 	mswsock.dll@0x96d2 	
4 	mswsock.dll 	mswsock.dll@0x266e 	
5 	mswsock.dll 	mswsock.dll@0x1b09 	
6 	ws2_32.dll 	NSPROVIDER::NSPLookupServiceNext 	
7 	ws2_32.dll 	NSPROVIDERSTATE::LookupServiceNext 	
8 	ws2_32.dll 	NSQUERY::LookupServiceNext 	
9 	ws2_32.dll 	WSALookupServiceNextW 	
10 	ws2_32.dll 	WSALookupServiceNextA 	
11 	ws2_32.dll 	getxyDataEnt 	
12 	ws2_32.dll 	gethostbyname 	
13 	nspr4.dll 	PR_GetHostByName 	nsprpub/pr/src/misc/prnetdb.c:722
14 	nspr4.dll 	pr_GetAddrInfoByNameFB 	nsprpub/pr/src/misc/prnetdb.c:1987
15 	nspr4.dll 	PR_GetAddrInfoByName 	nsprpub/pr/src/misc/prnetdb.c:2016
16 	xul.dll 	nsHostResolver::ThreadFunc 	netwerk/dns/src/nsHostResolver.cpp:884
17 	nspr4.dll 	_PR_NativeRunThread 	nsprpub/pr/src/threads/combined/pruthr.c:426
18 	xul.dll 	nsACString_internal::MutatePrep 	xpcom/string/src/nsTSubstring.cpp:179
19 	mozcrt19.dll 	_callthreadstartex 	obj-firefox/memory/jemalloc/src/threadex.c:348

Comment 1

[Wan-Teh Chang]

Thanks for the bug report.  Usually this kind of crash is caused
by a bug elsewhere.  I suggest that you start the investigation in
Necko (netwerk/dns/src/nsHostResolver.cpp).

There are no NSPR changes between Firefox 3.5 and 3.5.2.

There are only a small number of NSS changes between Firefox
3.5 and 3.5.1, and no NSS changes between Firefox 3.5.1 and 3.5.2.

Comment 2

[timeless]

So, some of these crashes actually have complete stack traces
http://code.google.com/p/chromium/issues/detail?id=17047#c0
indicates it's also a problem for chromium. We need someone to contact microsoft, we have hundreds of these.

Signature	wcslen
UUID	63d9af3c-442f-4cc2-8cd1-015972090809
Time 	2009-08-09 22:34:26.35625
Uptime	25
Last Crash	120683 seconds before submission
Product	Firefox
Version	3.5.2
Build ID	20090729225027
OS	Windows NT
OS Version	5.1.2600 Service Pack 2
CPU	x86
CPU Info	GenuineIntel family 15 model 4 stepping 9
Crash Reason	EXCEPTION_ACCESS_VIOLATION
Crash Address	0x0
User Comments	
Processor Notes 	
Crashing Thread
Frame 	Module 	Signature [Expand] 	Source
0 	msvcrt.dll 	wcslen 	
1 	mswsock.dll 	HostentBlob_WriteNameOrAlias 	
2 	mswsock.dll 	HostentBlob_CreateFromRecords 	
3 	mswsock.dll 	HostentBlob_WriteNameOrAlias 	
4 	mswsock.dll 	Rnr_DoDnsLookup 	
5 	mswsock.dll 	NSPLookupServiceNext 	
6 	ws2_32.dll 	NSPROVIDER::NSPLookupServiceNext 	
7 	ws2_32.dll 	NSPROVIDERSTATE::LookupServiceNext 	
8 	ws2_32.dll 	NSQUERY::LookupServiceNext 	
9 	ws2_32.dll 	WSALookupServiceNextW 	
10 	ws2_32.dll 	WSALookupServiceNextA 	
11 	ws2_32.dll 	getxyDataEnt 	
12 	ws2_32.dll 	gethostbyname 	
13 	nspr4.dll 	PR_GetHostByName 	nsprpub/pr/src/misc/prnetdb.c:722
14 	nspr4.dll 	pr_GetAddrInfoByNameFB 	nsprpub/pr/src/misc/prnetdb.c:1987
15 	nspr4.dll 	PR_GetAddrInfoByName 	nsprpub/pr/src/misc/prnetdb.c:2016
16 	xul.dll 	nsHostResolver::ThreadFunc 	netwerk/dns/src/nsHostResolver.cpp:884
17 	nspr4.dll 	_PR_NativeRunThread 	nsprpub/pr/src/threads/combined/pruthr.c:426
18 	xul.dll 	nsACString_internal::MutatePrep 	xpcom/string/src/nsTSubstring.cpp:179
19 	mozcrt19.dll 	_callthreadstartex 	obj-firefox/memory/jemalloc/src/threadex.c:348

Comment 3

[timeless]

so, i went through about 200 crashes from the url, and all but two are the signature i'm listing, i've filed bug 509365 for the other two.

Comment 4

[chris hofmann]

I looked though a sample of wcslen crashes from the 8/19

843 total crashes for wcslen on 20090819-crashdata.csv
408 start up crashes inside 3 minutes

os breakdown
 400 wcslen Windows NT 5.1.2600 Service Pack 3
 385 wcslen Windows NT 5.1.2600 Service Pack 2
  30 wcslen Windows NT 6.0.6001 Service Pack 1
   7 wcslen Windows NT 6.0.6000
   6 wcslen Windows NT 5.1.2600 Service Pack 1
   6 wcslen Windows NT 5.1.2600
   5 wcslen Windows NT 6.0.6002 Service Pack 2
   1 wcslen Windows NT 6.1.7100
   1 wcslen Windows NT 5.2.3790 Service Pack 2
   1 wcslen Windows NT 5.2.3790 Service Pack 1
   1 wcslen Windows NT 5.1.2600 Service Pack 3, v.3264

distribution of versions where the crash was found on 20090819-crashdata.csv
 494 Firefox 3.5.2
 251 Firefox 3.0.13
  21 Firefox 3.5
  19 Firefox 3.0.11
  11 Firefox 3.5.1
  11 Firefox 3.0.12
   9 Firefox 3.0
   7 Firefox 3.0.10
   6 Firefox 3.0.5
   4 Firefox 3.0.6
   3 Firefox 3.1b3
   3 Firefox 3.0.7
   2 Firefox 3.0.1

There is also some interesting stuff in the urls

an abnormaly high pct. of the crash urls are from the top level domain in turkey..

#crashes -  domain where the user might be when the crash happens
 210 http://www.google.com.tr
  45 http://tr.www.mozilla.com
  17 http://images.google.com.tr
   1 http://turkoloji.cu.edu.tr
  1 http://s12.travian.com.tr
   1 https://mail.etu.edu.tr
 and many more assorted...

and various other turkish sites or content

 2 http://turkcedublajsinema.bloggum.com/yazi

maybe some wild speculation but I wonder if that is somehow related to the hostname/dns look ups that are going on in the stack in comment 2 and there is some incompatibility we hit related to turkish sites...

Comment 5

[chris hofmann]

maybe erkan can help keep a look at for anecdotal reports of start up or other crashes in turkey were we might be hitting bad dns servers, sites with unusual host names that we might choke on, or some possible dns or content problem that might be related to a crash.

Comment 6

[Rail Aliiev [:rail]]

Add CC

Comment 7

[chris hofmann]

Its interesting to see there is a definite spike in the PR_GetHostByName crashes for this on 3.5.3 and 3.0.14 in the immediate days following their release.

   6 3.5.3 20090910-crashdata.csv
  69 3.5.3 20090911-crashdata.csv
  77 3.5.3 20090912-crashdata.csv
  80 3.5.3 20090913-crashdata.csv

   2 3.0.14 20090904-crashdata.csv
   1 3.0.14 20090908-crashdata.csv
   3 3.0.14 20090910-crashdata.csv
 137 3.0.14 20090911-crashdata.csv
 236 3.0.14 20090912-crashdata.csv
 187 3.0.14 20090913-crashdata.csv

volume for 3.0.x over the same period is much lower in comparison

   1 3.0.12 20090903-crashdata.csv
   4 3.0.12 20090907-crashdata.csv
   1 3.0.12 20090908-crashdata.csv

   1 3.0.13 20090903-crashdata.csv
   1 3.0.13 20090904-crashdata.csv
   1 3.0.13 20090905-crashdata.csv
   2 3.0.13 20090910-crashdata.csv

volume for 3.5.2 is much more steady and around the same level as 3.5.3

  81 3.5.2 20090901-crashdata.csv
  93 3.5.2 20090902-crashdata.csv
  76 3.5.2 20090903-crashdata.csv
  57 3.5.2 20090904-crashdata.csv
  94 3.5.2 20090905-crashdata.csv
  91 3.5.2 20090906-crashdata.csv
  50 3.5.2 20090907-crashdata.csv
  81 3.5.2 20090908-crashdata.csv
  55 3.5.2 20090909-crashdata.csv
  51 3.5.2 20090910-crashdata.csv
  33 3.5.2 20090911-crashdata.csv
  33 3.5.2 20090912-crashdata.csv
  28 3.5.2 20090913-crashdata.csv


This might indicate that the PR_GetHostByName problem is associated with new installs and/or updates.  Martin's comment 0 and filing of this bug was also right around the release time of 3.5.2 indicating a possible spike around that release.

Comment 8

[timeless]

my guess is that people have a problem (caused by LSPs), each time there's a new release available, they try it to see if it fixes the problem. it doesn't fix the problem, so they stop using firefox. this repeats with each release...

Comment 10

[Damon Sicore (:damons)]

Blocking 1.9.2+ as part of the CrashKill effort.

Comment 11

[Mike Beltzner [:beltzner, not reading bugmail]]

Marking all topcrash bugs as P2 (3.6 release blockers, but not 3.6b1 blockers)

Comment 12

[Justin Dolske [:Dolske]]

Indeed, looking through URLs associated with these crashes there are tons of Turkish sites. Including google.com.tr and tr.www.mozilla.com. I see these crashes with both short and long uptimes; but would have to have Lars run some DB queries to see if there's a strong correlation or not.

I can't reproduce the crash surfing around the various .tr URLs, next step would be to try changing locale and use a localized Firefox. I also can't help but wonder if it's possible there's a buggy or malicious Turkish ISP / DNS server that's twiddling some MS bug. Seems less likely, though, or we'd see it everywhere by now.

I happened to notice that one crash stack passed through cafeplusfilterhook.dll in an early frame, Googling this leads me to what appears to be Turkish filtering software, http://akinsoft.com.tr/. Probably just a coincidence, though, I don't see the module in other crash reports.

Also, dbaron noted that Turkish has interesting rules for converting between upper and lowercase letters (eg, http://en.wikipedia.org/wiki/Dotted_and_dotless_I). It's possible this could be involved in the crash, especially if buggy code is expecting the upper/lowercase transformation of a UTF8 string to have the same number of bytes.

Comment 13

[Rail Aliiev [:rail]]

AFAIK, TT (Türk Telekom) DNS cache servers use a filtering for blocking some sites. So it may be bugs in their DNS replies.

Comment 14

[(not currently active) Ted Mielczarek]

Might be interesting to see if we have a minidump available for this from the 24 hours of saved minidumps. We could at least verify that we're not passing bad data to gethostbyname then. Unfortunately without the heap data we probably can't find out exactly what is being passed.

Assuming we're not somehow passing bad data down, this has to be a Windows bug. It's quite deep in Windows system libraries, with no third-party code on the stack.

Comment 15

[(not currently active) Ted Mielczarek]

I acquired a dump, and I can say that we are passing a valid-looking char* to gethostbyname. Without heap memory there's not much else to be said.

Comment 16

[Justin Dolske [:Dolske]]

Yeah, I pulled a couple of minidumps last week and looked, seemed legit but the pointer wasn't to anything on the stack.

The 2nd frame's name (HostentBlob_CreateFromRecords) makes me suspect a bug with the way the OS's resolver is parsing the reply. Wonder if we can find someone who can reproduce this, record the network traffic with a packet sniffer, and forward on to Microsoft...

Comment 17

[(not currently active) Ted Mielczarek]

Getting a network trace or a full-memory minidump would definitely be ideal here. Could probably use some outreach to the Turkish community.

Comment 18

[chris hofmann]

they are on the cc.

Comment 19

[Rail Aliiev [:rail]]

> they are on the cc.

But most of them live out of Turkey or don't use Windows. :(

Comment 20

[Justin Dolske [:Dolske]]

Are there any Turkish Firefox user forums, where we might find people experiencing the problem and willing to help?

We might also try emailing people who have provided their email address in the crash report, although I'm wary of the odds of reaching advanced users that way (who could help with network analysis).

Comment 21

[(not currently active) Ted Mielczarek]

dolske: unfortunately we stopped collecting email addresses a while ago because we weren't actually using them. (bug 472357 unfortunately has yet to be fixed)

Comment 22

[Justin Dolske [:Dolske]]

(Clearing regression / regression-window-wanted, since this doesn't seem to actually be a regression, has unknown steps to reproduce, and is in a MS library.)

Comment 23

[Justin Dolske [:Dolske]]

(In reply to comment #19)

> But most of them live out of Turkey or don't use Windows. :(

Tomcat pointed out that you're on our Turkish L10N team and do QA, would you be able to try and reproduce this?

Comment 24

[Axel Hecht [:Pike]]

CCing Kadir, he might have contacts within Turkey that can help?

Comment 25

[Tim Riley [:timr]]

I am going to pile on a bit.  Erkan - As the Turkish locale owner, can you help find reproducible test cases from the Turkish sites.  Or are there others that you know that might be able to help?

Comment 26

[Kadir Topal [:atopal]]

Well, most of my relatives in Turkey do use Firefox, but they aren't exactly computer experts. And with unknown steps to reproduce this bug I guess they won't be of much help.

Comment 27

[Rail Aliiev [:rail]]

(In reply to comment #23)
> (In reply to comment #19)
> 
> > But most of them live out of Turkey or don't use Windows. :(
> 
> Tomcat pointed out that you're on our Turkish L10N team and do QA, would you be
> able to try and reproduce this?

Well, I'm a part of Turkish L10N team, but live not in Turkey. IIRC, this issue is reproducible only in Turkey.

I can try to reproduce this bug within a virtual machine outside of Turkey. If it is not repoducible I'll try get help from other community members...

Comment 28

[Justin Dolske [:Dolske]]

Belated update: last week we made contact with the Microsoft folks and have a case open for this. I suspect progress will be slow, though, until we're able to collect additional data from a user experiencing the problem.

Comment 29

[Justin Dolske [:Dolske]]

I had our Socorro guys run a query to dump all reports for this crash within the last 3 months where the user added a comment or provided their email address (bug 525962)... Unfortunately none of the 42 such submissions contained an email address in either field. [We might try running this query again later, because it was only recently that we started saving submitted email addresses.]

bp-f0f54cac-2bcd-4b35-8cde-7262a2091018 was one of the few that contained a comment that was (1) of significant length (2) in English (3) and wasn't cursing us. :-) 

"this version of Firefox is really makes me messed up, i have never had such a redicilous problems ever, it's happenning all the time after i make google search i touch search button and Boom Mozilla crashes i have disabled  ALL Extensions and Plug- in but no way what's wrong with this verison of Firefox :S, i haev never had problems with this browser..."

That does suggest that this is happening repeatedly to users, and that the crashes on Google are not just an artifact of out sometimes-broken reporting of the URL that the user crashed on.

One curious thing about the crash description is that it's a bit strange to be doing a DNS resolution for a search submission. Maybe a google.com --> google.tr transition? Or other content in the page?

This also makes me wonder if DNS prefetching could be exasperating the problem. If so, turning it off in Turkish localized builds might help mitigate these crashes (though I don't know how many users are actually using the localized version).

Comment 30

[Rail Aliiev [:rail]]

I've tried to reproduce this bug but without luck.

For those who want to investigate the DNS replies from servers used for most of Internet users in Turkey, here are DNS servers of TTNet (the only, AFAIK, uplink owner):

195.175.39.39
195.175.39.40

They work even if you are not within their network. And don't be surprised if you cannot reach youtude.com. :D

Comment 31

[chris hofmann]

I wonder if there is any kind of tor debugging mode that might force traffic through Turkey that could also help out in others being able to participate in debuging this?

Comment 32

[Damon Sicore (:damons)]

In talks with Microsoft, still need to repro, so unblocking.

Comment 33

[Wan-Teh Chang]

We collected some info from Google Chrome crashes in getaddrinfo.
We can't establish a relation to the Turkish locale.  Our findings
are summarized at
http://code.google.com/p/chromium/issues/detail?id=22083#c16

Comment 34

[Justin Dolske [:Dolske]]

(Unassigning, I'm not actively tracking this)

Comment 35

[Sheila Mooney]

Still a valid crash but appears at #170 on 8.0. Removing the top crash keyword.

Comment 38

[Scoobidiver (away)]

Based on recent comments, I doubt it's still fully correlated to Turkish users.

I see Vietnamese comments confirmed by correlations:
     41% (23/56) vs.   1% (964/136061) UKHook40.dll (Vietnamese keyboard)
Russian ones confirmed also by correlations:
     11% (6/56) vs.   3% (4511/136061) [e-mail]
      9% (5/56) vs.   3% (4437/136061) [e-mail] (Yandex.Bar, https://addons.mozilla.org/addon/3495)
Turkish and English ones.

More reports at:
https://crash-stats.mozilla.com/report/list?signature=wcslen+|+HostentBlob_WriteNameOrAlias