Open Bug 530074 Opened 10 years ago Updated 1 year ago

Crash [@ StrChrIA ] from winsock and shlwapi.dll (LSP, evil *x86.dll module)

Categories

(Core :: Networking: HTTP, defect, P2, critical)

x86
Windows 7
defect

Tracking

()

Tracking Status
firefox14 + ---
firefox16 - ---
firefox17 - ---

People

(Reporter: jimm, Assigned: mayhemer)

References

(Blocks 1 open bug)

Details

(Keywords: crash, Whiteboard: [crashkill][crashkill-outreach][crashkill-blocklist][explosive?][unactionable][necko-triaged])

Crash Data

Attachments

(1 file)

0  	shlwapi.dll  	StrChrIA  	
1 	shlwapi.dll 	StrStrIA 	
2 	ws2_32.dll 	WSARecv (winsock)

http://crash-stats.mozilla.com/report/index/0db839af-8bc4-4d5f-92e3-597482091120

Processor notes: WARNING: "[u'6']" is deficient as a name and version for an addon; WARNING: "[u'2']" is deficient as a name and version for an addon; WARNING: "[u'48']" is deficient as a name and version for an addon
Summary: Top crash in StrChrIA → Crash [@StrChrIA ]
Probably an LSP
Severity: normal → critical
Summary: Crash [@StrChrIA ] → Crash [@ StrChrIA ]
Adding the space in the title so breakpad picks up the bug.
Summary: Crash [@ StrChrIA ] → Crash [ @ StrChrIA ]
Signature	StrChrIA
UUID	e60656f9-e02d-4563-8782-7563a2091124
Time 	2009-11-24 09:05:14.693524
Uptime	54527
Last Crash	134624 seconds before submission
Product	Firefox
Version	3.5.3
Build ID	20090824101458
Branch	1.9.1
OS	Windows NT
OS Version	5.1.2600 Service Pack 2
CPU	x86
CPU Info	GenuineIntel family 6 model 14 stepping 8
Crash Reason	EXCEPTION_ACCESS_VIOLATION
Crash Address	0xe564000
User Comments	
Processor Notes 	
Crashing Thread
Frame 	Module 	Signature [Expand] 	Source
0 	shlwapi.dll 	StrChrIA 	
1 	shlwapi.dll 	StrStrIA 	
2 		@0x1066cfb 	
3 		@0x1066d7b 	
4 		@0x1065c2e 	
5 	ws2_32.dll 	WSARecv 	
6 	wsock32.dll 	recv 	
7 	nspr4.dll 	_PR_MD_RECV 	nsprpub/pr/src/md/windows/w95sock.c:327
8 	nspr4.dll 	SocketRead 	nsprpub/pr/src/io/prsocket.c:657
9 	xul.dll 	nsSocketInputStream::Read 	netwerk/base/src/nsSocketTransport2.cpp:353
10 	xul.dll 	nsHttpConnection::OnWriteSegment 	netwerk/protocol/http/src/nsHttpConnection.cpp:632
11 	xul.dll 	nsHttpTransaction::WritePipeSegment 	netwerk/protocol/http/src/nsHttpTransaction.cpp:503
12 	xul.dll 	nsPipeOutputStream::WriteSegments 	xpcom/io/nsPipe3.cpp:1137
13 	xul.dll 	nsHttpTransaction::WriteSegments 	netwerk/protocol/http/src/nsHttpTransaction.cpp:529
14 	xul.dll 	nsHttpConnection::OnSocketReadable 	netwerk/protocol/http/src/nsHttpConnection.cpp:664
15 	xul.dll 	nsHttpConnection::OnInputStreamReady 	netwerk/protocol/http/src/nsHttpConnection.cpp:762
16 	xul.dll 	nsSocketInputStream::OnSocketReady 	netwerk/base/src/nsSocketTransport2.cpp:256
17 	xul.dll 	nsSocketTransport::OnSocketReady 	netwerk/base/src/nsSocketTransport2.cpp:1519
18 	xul.dll 	nsSocketTransportService::DoPollIteration 	netwerk/base/src/nsSocketTransportService2.cpp:674
19 	xul.dll 	nsSocketTransportService::OnProcessNextEvent 	netwerk/base/src/nsSocketTransportService2.cpp:538
20 	xul.dll 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:497
21 	xul.dll 	NS_ProcessPendingEvents_P 	obj-firefox/xpcom/build/nsThreadUtils.cpp:180
22 	xul.dll 	NS_ProcessNextEvent_P 	obj-firefox/xpcom/build/nsThreadUtils.cpp:227
23 	xul.dll 	nsSocketTransportService::Run 	netwerk/base/src/nsSocketTransportService2.cpp:581
24 	xul.dll 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:510
25 	xul.dll 	nsThread::ThreadFunc 	xpcom/threads/nsThread.cpp:254
26 	nspr4.dll 	_PR_NativeRunThread 	nsprpub/pr/src/threads/combined/pruthr.c:426
27 	nspr4.dll 	pr_root 	nsprpub/pr/src/md/windows/w95thred.c:122
28 	mozcrt19.dll 	_callthreadstartex 	obj-firefox/memory/jemalloc/src/threadex.c:348
29 	mozcrt19.dll 	_threadstartex 	obj-firefox/memory/jemalloc/src/threadex.c:326
30 	kernel32.dll 	BaseThreadStart 	

Filename 	Version 	Debug Identifier 	Debug Filename
NPSWF32.dll 	10.0.32.18 	C569605C15E5448BBFD9E1FE262649B61 	NPSWF32.pdb
mdnsNSP.dll 	1.0.6.2 	E82F22B11854424A8DAE403E0F7A62A91 	mdnsNSP.pdb
ccL40.dll 	104.0.1.17 	05AB7705112C4149AC4870E12D01E6F81 	ccL40.pdb

I don't think any of these are related but i'd love to be proven wrong. Frames 2-4 scare me.

Signature	StrChrIA
UUID	abf3bc64-c231-431a-836a-fd47b2091122
Time 	2009-11-22 09:10:13.599272
Uptime	464
Last Crash	408839 seconds before submission
Product	Firefox
Version	3.5.3
Build ID	20090824101458
Branch	1.9.1
OS	Windows NT
OS Version	5.1.2600 Service Pack 2
CPU	x86
CPU Info	GenuineIntel family 15 model 4 stepping 3
Crash Reason	EXCEPTION_ACCESS_VIOLATION
Crash Address	0x5c7c000
User Comments	
Processor Notes 	WARNING: No 'client_crash_date' could be determined from the Json file
Crashing Thread
Frame 	Module 	Signature [Expand] 	Source
0 	shlwapi.dll 	StrChrIA 	
1 	shlwapi.dll 	StrStrIA 	
2 	DA475474.x86.dll 	DA475474.x86.dll@0x28a3 	

I like this one because we can see a bad guy at the bottom.

Signature	StrChrIA
UUID	ecb0fab6-2fae-40e0-8dd2-704722091122
Time 	2009-11-22 23:05:57.162820
Uptime	4641
Last Crash	4644 seconds before submission
Product	Firefox
Version	3.5.3
Build ID	20090824101458
Branch	1.9.1
OS	Windows NT
OS Version	5.1.2600 Service Pack 3
CPU	x86
CPU Info	GenuineIntel family 15 model 4 stepping 9
Crash Reason	EXCEPTION_ACCESS_VIOLATION
Crash Address	0x118e2000
User Comments	
Processor Notes 	WARNING: "[u'6']" is deficient as a name and version for an addon; WARNING: "[u'2']" is deficient as a name and version for an addon; WARNING: "[u'41']" is deficient as a name and version for an addon
Crashing Thread
Frame 	Module 	Signature [Expand] 	Source
0 	shlwapi.dll 	StrChrIA 	
1 	shlwapi.dll 	StrStrIA 	
2 	7129F7E2.x86.dll 	7129F7E2

again, something that looks like a bad guy.

Signature	StrChrIA
UUID	6f42b825-ec68-486e-9b51-59a422091119
Time 	2009-11-19 06:36:28.375838
Uptime	856
Last Crash	865 seconds before submission
Product	Firefox
Version	3.5.3
Build ID	20090824101458
Branch	1.9.1
OS	Windows NT
OS Version	6.1.7100
CPU	x86
CPU Info	GenuineIntel family 6 model 14 stepping 12
Crash Reason	EXCEPTION_ACCESS_VIOLATION
Crash Address	0x890d000
User Comments	
Processor Notes 	
Crashing Thread
Frame 	Module 	Signature [Expand] 	Source
0 	shlwapi.dll 	StrChrIA 	
1 	shlwapi.dll 	StrStrIA 	
2 	avsda.dll 	avsda.dll@0x245d 	

avsda.dll 	9.0.1.1 	B9A99DF4825F41B781F82B2E9AD4BAC91 	avsda.pdb

this is Avira's LSP, so I'm right in at least one instance :)

this is a semi random sampling, i think i skipped only one crash from my sample because it wasn't telling me anything more than jim's original
Summary: Crash [ @ StrChrIA ] → Crash [@ StrChrIA ] from winsock (LSP or evil module)
for some reason this signature has gone crazy over the last few days

report for the stack signature StrChrIA 
64  crashes for StrChrIA on  20091201-crashdata
65  crashes for StrChrIA on  20091202-crashdata
52  crashes for StrChrIA on  20091203-crashdata
53  crashes for StrChrIA on  20091204-crashdata
49  crashes for StrChrIA on  20091205-crashdata
45  crashes for StrChrIA on  20091206-crashdata
53  crashes for StrChrIA on  20091207-crashdata
255  crashes for StrChrIA on  20091208-crashdata
430  crashes for StrChrIA on  20091209-crashdata
573  crashes for StrChrIA on  20091210-crashdata
623  crashes for StrChrIA on  20091211-crashdata
618  crashes for StrChrIA on  20091212-crashdata
654  crashes for StrChrIA on  20091213-crashdata
628  crashes for StrChrIA on  20091214-crashdata
684  crashes for StrChrIA on  20091215-crashdata
737  crashes for StrChrIA on  20091216-crashdata

currently ranked around #27 and up 140 slots in the 7 day report.
Whiteboard: [crashkill]
shlwapi.dll is microsoft library which contains functions for UNC and URL paths, registry entries, and color settings.

2009 12 08 is patch tuesday.  wonder if there is any connection and the new update had tickled a volume increase or a new bug?
Whiteboard: [crashkill] → [crashkill][crashkill-outreach][explosive?]
all     232213  737     0.00317381
3.0.15  32264   120     0.00371932
3.5.5   93452   311     0.00332791
3.6b4   21927   90      0.00410453
3.6b3   675             0
3.6b2   849     3       0.00353357
3.6b1   2304    1       0.000434028

all releases
   3 3.0.1
   1 3.0.10
   1 3.0.12
   1 3.0.14
 120 3.0.15
  42 3.0.16
   1 3.0.7
   2 3.0.8
   1 3.1b2
   4 3.5
   5 3.5.2
   1 3.5.3
   2 3.5.4
 311 3.5.5
 148 3.5.6
   1 3.6b1
   3 3.6b2
  90 3.6b4
crash urls are pretty widely and uniformly distributed over popular browsing sites

domains of sites
  56 \N//
  55 http://apps.facebook.com
  39 http://www.facebook.com
  34 about:blank//
  18 http://messaging.myspace.com
  18 http://home.myspace.com
  17 http://www.myspace.com
  16 http://viewmorepics.myspace.com
  15 http://www.google.com
  13 http://www.tagged.com
  11 http://profile.myspace.com
  10 http://twitter.com
 < long tail snipped>
Signature	StrChrIA
UUID	1c3122b0-5d90-427f-ad29-d52fc2091217
Time 	2009-12-17 22:29:50.830579
Uptime	5379
Last Crash	38531 seconds before submission
Product	Firefox
Version	3.5.3
Build ID	20090824101458
Branch	1.9.1
OS	Windows NT
OS Version	5.1.2600 Service Pack 3
CPU	x86
CPU Info	GenuineIntel family 15 model 4 stepping 3
Crash Reason	EXCEPTION_ACCESS_VIOLATION
Crash Address	0xe2a3000
User Comments	
Processor Notes 	WARNING: No 'client_crash_date' could be determined from the Json file
Related Bugs

Crashing Thread
Frame 	Module 	Signature [Expand] 	Source
0 	shlwapi.dll 	StrChrIA 	
1 	shlwapi.dll 	StrStrIA 	
2 	B3180C00.x86.dll 	B3180C00.x86.dll@0x28a3 	
Thread 3
Frame 	Module 	Signature [Expand] 	Source
0 	ntdll.dll 	KiFastSystemCallRet 	
1 	ntdll.dll 	ZwRemoveIoCompletion 	
2 	B3180C00.x86.dll 	B3180C00.x86.dll@0x4aab 	
3 	kernel32.dll 	BaseThreadStart 	
Thread 4
Frame 	Module 	Signature [Expand] 	Source
0 	ntdll.dll 	KiFastSystemCallRet 	
1 	ntdll.dll 	NtReplyWaitReceivePortEx 	
2 	B3180C00.x86.dll 	B3180C00.x86.dll@0x1655 	
3 	kernel32.dll 	BaseThreadStart 	

Can we blocklist any file named *.x86.dll?
Whiteboard: [crashkill][crashkill-outreach][explosive?] → [crashkill][crashkill-outreach][crashkill-blocklist][explosive?]
more discussion of blocking *x86.dll over in bug 516112
Depends on: 516112
Summary: Crash [@ StrChrIA ] from winsock (LSP or evil module) → Crash [@ StrChrIA ] from winsock (LSP or evil *x86.dll module)
recently running at about 600-850 crashes per day.

date StrChrIAcrashes
20100101-crashdata 601 StrChrIA
20100102-crashdata 686 StrChrIA
20100103-crashdata 798 StrChrIA
20100104-crashdata 842 StrChrIA
20100105-crashdata 824 StrChrIA
20100106-crashdata 854 StrChrIA
20100107-crashdata 877 StrChrIA
20100108-crashdata 721 StrChrIA
20100109-crashdata 728 StrChrIA
also more dicussion of Crashes [@ shlwapi.dll over in Bug 556178
Depends on: 556178
Summary: Crash [@ StrChrIA ] from winsock (LSP or evil *x86.dll module) → Crash [@ StrChrIA ] from winsock and shlwapi.dll (LSP or evil *x86.dll module)
Crash Signature: [@ StrChrIA ]
Still pretty high volume - sitting at #45 for 8.0. Leaving the top crash keyword for now. Is there something we can block here? Doesn't seem very actionable.
This was a bunch lower in 9.0.1. Outside the top 100. I am going to remove the topcrash keyword.
Keywords: topcrash
It's #16 top browser crasher in 12.0, #29 in 13.0b4, and #11 in 14.0a2.
Keywords: topcrash
Version: 1.9.2 Branch → Trunk
Still high - sitting at #29 on the top crash list.
It's #24 top browser crasher in 13.0.1, but up to #11 in 14.0b10 and #6 in 14.0b11.

There are no obvious correlations in 13.0.1, but in 14.0 Beta it's correlated to BitDefender 2012:
StrChrIA|EXCEPTION_ACCESS_VIOLATION_READ (730 crashes)
     55% (400/730) vs.   1% (462/65994) BdProvider.dll
          0% (1/730) vs.   0% (1/65994) 16.0.14.1139
          6% (45/730) vs.   0% (57/65994) 16.15.0.1213
          4% (28/730) vs.   0% (44/65994) 16.16.0.1317
         45% (325/730) vs.   1% (356/65994) 16.16.0.1339
          0% (1/730) vs.   0% (4/65994) 16.17.0.1350
     51% (369/730) vs.   1% (533/65994) avcuf32.dll
          0% (0/730) vs.   0% (4/65994) 3.8.5601.4190
          0% (0/730) vs.   0% (3/65994) 3.9.6208.4190
         51% (369/730) vs.   1% (526/65994) 3.9.6339.4190

Here are the first frames of the matching stack trace:
Frame 	Module 	Signature 	Source
0 	shlwapi.dll 	StrChrIA 	
1 	shlwapi.dll 	StrStrIA 	
2 	BdProvider.dll 	BdProvider.dll@0x2777 	
3 	BdProvider.dll 	BdProvider.dll@0xe444 	
4 	ws2_32.dll 	send 	
5 	nspr4.dll 	SocketWrite 	nsprpub/pr/src/io/prsocket.c:701
6 	xul.dll 	nsSocketOutputStream::Write 	netwerk/base/src/nsSocketTransport2.cpp:587
7 	xul.dll 	nsHttpConnection::OnReadSegment 	netwerk/protocol/http/nsHttpConnection.cpp:1190
8 	xul.dll 	nsHttpPipeline::ReadFromPipe 	netwerk/protocol/http/nsHttpPipeline.cpp:682

More reports at:
https://crash-stats.mozilla.com/report/list?signature=StrChrIA
Component: Networking → Networking: HTTP
Summary: Crash [@ StrChrIA ] from winsock and shlwapi.dll (LSP or evil *x86.dll module) → Crash [@ StrChrIA ] from winsock and shlwapi.dll (LSP, evil *x86.dll module, or BitDefender)
This bug was #16 in 12.0 and has risen a bit in FF14. I think our best bet is to do outreach here and try the URLs in QA. Including Kev here and adding qawanted. Also tentatively tracking for release, although this wouldn't block.
Total Count 	URL
84 	http://www.facebook.com/
57 	about:blank
11 	https://www.facebook.com/login.php?login_attempt=1
11 	http://www.tumblr.com/dashboard
10 	http://www.facebook.com/?ref=tn_tnmn
10 	http://www.meetme.com/apps/home
9 	about:newtab
6 	http://www.yahoo.com/
4 	http://www.facebook.com/?ref=logo
4 	http://www.google.com/
4 	http://www.meetme.com/
4 	https://www.facebook.com/
4 	http://www.expressz.hu/index.do?s_=1&so_ns_rid=1&x=regions%3A9&cid=2909&parentRe
3 	http://messages.meetme.com/
3 	http://www.youtube.com/
3 	http://www.odnoklassniki.ru/
3 	http://www.milliyet.com.tr/
3 	http://www.crtinv.com/products/ADDictThing/default/getData.html?d=http%3A%2F%2Fw
3 	https://www.google.co.uk/
3 	http://www.cucirca.com/2008/06/19/one-tree-hill-season-5-episode-6-dont-dream-it
3 	http://www.freelove.hu/index.php?id=14&sub_id=13
2 	http://www.ilsole24ore.com/?refresh_ce
2 	http://www.lequipe.fr/
2 	http://www.theblaze.com/
2 	http://www.tumblr.com/inbox
2 	http://apps.facebook.com/warcommander/?fb_source=bookmark_apps&ref=bookmarks&cou
2 	http://www.youtube.com/inbox?feature=mhee&folder=messages
2 	http://facebook.com/
2 	http://www.transfermarkt.de/de/1-fsv-mainz-05/startseite/verein_39.html
2 	http://www.worldstarhiphop.com/videos/
2 	http://yahoo.com/
2 	http://www.lrytas.lt/
2 	http://www.tuenti.com/#m=Home&func=index
2 	http://cgi5.ebay.de/ws/eBayISAPI.dll
2 	http://apps.facebook.com/avengersalliance/?fb_source=bookmark_apps&ref=bookmarks
2 	http://apps.facebook.com/pioneertrail/?fb_source=bookmark_apps&ref=bookmarks&cou
2 	http://apps.facebook.com/playslingo/?fb_source=bookmark_apps&ref=bookmarks&count
2 	http://www.kansascity.com/sports/
2 	http://www.deviantart.com/
2 	http://www.gameduell.fr/gd/singleplayer03GameStart.do
2 	https://ssl.meetme.com/login
2 	http://www.profblog.org/
2 	http://www.speedtest.net/
2 	http://xfinity.comcast.net/
2 	http://globoesporte.globo.com/
Keywords: needURLs
(In reply to Scoobidiver from comment #17)
> There are no obvious correlations in 13.0.1, but in 14.0 Beta it's
> correlated to BitDefender 2012

This keeps to be the case, and I think that's something we should investigate, and possibly also to outreach to them.
I've been trying to reproduce this problem on Fx14.0b12 using the URLs listed here, doing different kinds of interactions, while having installed the BitDefender antivirus software as well as their couple of add-ons listed in AMO. No luck so far.
https://crash-stats.mozilla.com/report/list?signature=StrCmpNIA is another signature which I see in early 14.0.1 data that seems to be correlated to Bit Defender as well.
Addon correlations from Firefox 14.0.1: 

StrChrIA|EXCEPTION_ACCESS_VIOLATION_READ (1047 crashes)
     25% (263/1047) vs.  10% (6643/66763) {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d} (Adblock Plus, 
https://addons.mozilla.org/addon/1865)
          0% (0/1047) vs.   0% (14/66763) 1.3.10
          0% (0/1047) vs.   0% (2/66763) 1.3.3
          0% (0/1047) vs.   0% (2/66763) 1.3.5
          0% (0/1047) vs.   0% (1/66763) 2.0.1
          0% (0/1047) vs.   0% (2/66763) 2.0.2
          0% (4/1047) vs.   0% (131/66763) 2.0.3
          0% (0/1047) vs.   0% (14/66763) 2.1
         25% (259/1047) vs.  10% (6474/66763) 2.1.1
          0% (0/1047) vs.   0% (1/66763) 2.1.2a.3522
          0% (0/1047) vs.   0% (1/66763) 2.1.2a.3523
          0% (0/1047) vs.   0% (1/66763) 2.1.2b.3528
Crash Signature: [@ StrChrIA ] → [@ StrChrIA ] [@ StrCmpNIA]
I was able to repro this bug once on my machine. At the time of the crash a dialog came up asking me to install a root certificate. I will try to get a better set of steps or see if I can crash again.

https://crash-stats.mozilla.com/report/index/bp-34503c32-9845-4f2e-aa0c-d2ff12120720
(In reply to Marcia Knous [:marcia] from comment #24)
> I was able to repro this bug once on my machine. At the time of the crash a
> dialog came up asking me to install a root certificate. I will try to get a
> better set of steps or see if I can crash again.
> 
> https://crash-stats.mozilla.com/report/index/bp-34503c32-9845-4f2e-aa0c-
> d2ff12120720

Any updates here Marcia?
No luck reproducing on the same machine. It seems that the dialog that comes up asking me to save the root certificate doesn't work that well because I keep getting the prompt, but no crash.

(In reply to Alex Keybl [:akeybl] from comment #25)
> (In reply to Marcia Knous [:marcia] from comment #24)
> > I was able to repro this bug once on my machine. At the time of the crash a
> > dialog came up asking me to install a root certificate. I will try to get a
> > better set of steps or see if I can crash again.
> > 
> > https://crash-stats.mozilla.com/report/index/bp-34503c32-9845-4f2e-aa0c-
> > d2ff12120720
> 
> Any updates here Marcia?
http://forum.bitdefender.com/index.php?showtopic=35932 seems to have some information, reading through it now.
I have posted in that forum asking for more information from the users that are hitting the crash.
That thread indicates it's about BD 2013 and is fixed in 15.0 Beta.
It still happens in 15.0 Beta but less often according to crash stats as it's #28 top browser crasher in 13.0.1, #10 in 14.0.1 and #20 in 15.0b1 and correlations per module:
* 13.0.1: no correlation to BdProvider.dll
* 14.0.1:
     63% (1343/2137) vs.   1% (1494/151974) BdProvider.dll
          0% (1/2137) vs.   0% (1/151974) 1.0.0.177
          0% (1/2137) vs.   0% (1/151974) 16.0.14.1139
          1% (30/2137) vs.   0% (37/151974) 16.15.0.1213
          2% (33/2137) vs.   0% (39/151974) 16.16.0.1317
         24% (523/2137) vs.   0% (576/151974) 16.16.0.1339
         35% (755/2137) vs.   1% (840/151974) 16.17.0.1350
* 15.0:
     38% (73/192) vs.   0% (103/48561) BdProvider.dll
          1% (1/192) vs.   0% (1/48561) 16.0.14.1139
          0% (0/192) vs.   0% (1/48561) 16.15.0.1213
          2% (4/192) vs.   0% (5/48561) 16.16.0.1317
          7% (13/192) vs.   0% (22/48561) 16.16.0.1339
         29% (55/192) vs.   0% (74/48561) 16.17.0.1350

It's likely related to other HTTP networking crashes in nsHttpConnection::OnReadSegment that also spiked in 14.0: bug 763261, bug 763385, bug 763386, bug 763392, bug 763393, bug 763395, and bug 763398.
https://crash-stats.mozilla.com/report/index/bp-c61a45ae-da1b-4d4c-bdcb-34e082120725 - I crashed on another machine in the lab. 

One thing I consistently note is that I keep getting a Bit Defender prompt to install a certificate. I click OK to close Firefox and install it, but I keep consistently getting the prompt thereafter.
The Win 7 lab machine seems to crash much more often than the other machine I tested on. This machine has crashed at least 6-7 times in the last 15 minutes. Usually when I crash I do see the same dialog referenced in Comment 30.

On this Win 7 Machine I am running Bitdefender Internet Security 2013/Firefox 14.0.1.

I noticed when I was on the liveleak.com site and started opening a bunch of different videos, I started crashing more often.
I got the attached after a recent crash. You can see the dialog I was referring to in previous comments.

I can get this crash to reproduce fairly reliably by just clicking on links from liveleak.com.
(In reply to Marcia Knous [:marcia] from comment #32)
> I can get this crash to reproduce fairly reliably by just clicking on links
> from liveleak.com.
People in the thread say it crashes less often with 15.0 Beta. Can you give it a try?
I took a look at the win 7 machine in the lab running firefox in a debugger. Everything looks fine on our end of things, I suspect this is caused by a bug in bitdefender.
(In reply to Nick Hurley [:hurley] from comment #34)
> I suspect this is caused by a bug in bitdefender.
One user says it doesn't happen in Fx 12.0 (see http://forum.bitdefender.com/index.php?s=&showtopic=35932&view=findpost&p=150695), other users say it's with Fx 15.0 Beta (see comment 29). So it's in Firefox 14.0.1.
(Bitdefender dev here)
According to QA we fixed a bug very similar to mozilla bug 530074, we are putting it on update next Monday. I honestly appreciate everyone's help in identifying this.
harjoc - I appreciate the communication here.
If this still happens in connection with Bitdefender, please check that you have the latest relevant BdProvider.dll -- 16.18.0.1406 -- which was pushed this week. The security suite should update it automatically. 

I still see reports on crash-stats with 16.17.0.1350 which is old (and other unrelated LSP offenders) fwiw.
harjoc - can you give us any idea of the kind of behavior that tickled the BD bug? Reports seem to indicate different firefox revs had different levels of crashes, so I'm curious from a defensive pov.
I am experiencing constant crashing while using latest Bit Defender Total Security 2013 - 16.18.0.1406 and FireFox 14.01 running on windows 7.

From the looks of it so far the crash seems to happen when Antiphishing is enabled. What it does is scan every link on the page and puts a green check mark next to it. The crash seems to happen most often at ajax heavy sites.
Just got another crash while doing a Quick Virus Scan as well.
Now I crashed while not even focused on the browser. It seems when I narrow it down to something, something else causes it to crash :(
E, can you provide your crash ID from about:crashes?
McManus, I asked for more info on how to reproduce, hopefully E gives us a crash ID in the mean time.
A SUMO User who seems to be experiencing this issue. https://support.mozilla.org/en-US/questions/933874

QA, do we need to do 1:1 with them? I can step in and help out.
E, thank you for the crash ids. I looked at them all and you apparently have BdProvider.dll 16.17.0.1350 (not the latest version).

I just took the liberty earlier to scrape all the crashes for "StrChrIA" for Firefox-14.0.1 from yesterday (2198 crashes).

Apparently the bug reproduced on Vista and Win7, so I'm pulling the raw data for NT6.0 and NT 6.1, and looking at all the BdProvider.dll occurences (700 raw reports so far):

      7 BdProvider.dll|16.15.0.1213
      9 BdProvider.dll|16.16.0.1317
    112 BdProvider.dll|16.16.0.1339
    293 BdProvider.dll|16.17.0.1350
    421 total

So, 0 hits for 16.18.0.1406 (on Vista and Win7).

There were 236 crashes on Firefox 15.0b2 yesterday which I guess are also mostly with BdProvider 16.17.

As most users update their security product, the Bitdefender-related crash count will probably go back towards zero.
How do we update it? The reason I ask is when I go to about page on my security it says:

16.18.0.1406

but if I go to details on the bdprovider.dll, it is as you said its BdProvider 16.17
Ok nvm, all it took was a restart of the computer. Weird that Bit Defender does not prompt to restart after upgrade.

Anyways I am turning on the phishing filter and will report how things go.
My Win 7 lab machine is still getting crashes, and the BitDefender about: page shows I am running 16.18.0.1406.  Here is my latest crash report:

https://crash-stats.mozilla.com/report/index/bp-7891fa14-b3fe-4bc8-aa30-fcd4b2120806

I did restart the computer after the update was applied. The dll version from the module section of the crash shows 16.16.0.1339 and so does the dll when I search for it on my computer.

So somehow even after a restart my machine did not get updated. If I check for updates manually there are no updates found.
(In reply to Marcia Knous [:marcia] from comment #50)
> My Win 7 lab machine is still getting crashes, and the BitDefender about:
> page shows I am running 16.18.0.1406.
This dll version is not widespread according to crash stats in 14.0.1:
 1.0.0.177	0.65%
 16.15.0.1213	1,20%
 16.16.0.1317	1.42%
 16.16.0.1339	31.70%
 16.17.0.1350	64.16%
 16.18.0.1406	0.87%
So there's something wrong with the BD update process.
marcia, update failures happen because
 - the file could not be overwritten (there's an .upd file next to the .dll), or
 - the update download failed

In both cases there should be a related message in the Events page.

If not, the only suggestion so far I got from QA was to restart (I know...). I'll ask for more info if things still don't work.
Scoobidiver, it's not clear if 16.18.* is not widespread in crashes because people haven't updated to it, or because it fixes the problem.

From the 1392 BdProvider-related reports from last Friday I saw 0 reports with 16.18.*. 

I've been trying to get some numbers from the update team to see if people are having problems updating. I'll share them once I have them.
It's #2 top browser crasher in 16.0 and is slightly correlated to Babylon, Apple's Bonjour and Avira Antivirus:
  StrChrIA|EXCEPTION_ACCESS_VIOLATION_READ (394 crashes)
     27% (106/394) vs.   6% (3754/63152) browsemngr.dll
     34% (135/394) vs.  14% (9152/63152) mdnsNSP.dll
     17% (68/394) vs.   2% (1459/63152) avsda.dll
  StrCmpNIA|EXCEPTION_ACCESS_VIOLATION_READ (46 crashes)
     22% (10/46) vs.   2% (1459/63152) avsda.dll
     24% (11/46) vs.   6% (3754/63152) browsemngr.dll
     30% (14/46) vs.  14% (9152/63152) mdnsNSP.dll
Can we block all version of the offending dll prior to 16.18.*? Or are LSPs immune from that?
(In reply to Jim Mathies [:jimm] from comment #55)
> Can we block all version of the offending dll prior to 16.18.*?
It's no longer correlated to bdprovider.dll (the BD update process is probably done for every users) so blocking those versions won't help to reduce the crash volume.
Summary: Crash [@ StrChrIA ] from winsock and shlwapi.dll (LSP, evil *x86.dll module, or BitDefender) → Crash [@ StrChrIA ] from winsock and shlwapi.dll (LSP, evil *x86.dll module)
(In reply to Jim Mathies [:jimm] from comment #55)
> Can we block all version of the offending dll prior to 16.18.*? Or are LSPs
> immune from that?

I don't know of any way we'd have to block LSPs (and we don't even do any summary/correlation reports on them, so we don't have good data if a well-identifiable LSP is behind this). Also, those DLL correlations in comment #54 are weak enough that they're not pointing to anything specific, they just say that people running into this are more likely to have those things installed - but no smoking gun here so far.
Leaving this nominated - the crash volume is too low to know for sure whether this will be a top issue.
Without hangs, it's #2 top crasher in 16.0 and #4 in 15.0.1 so it doesn't seem related to a regression in 16.0 although it spiked recently.

Today's correlations are similar to those of comment 54:
*15.0.1:
  StrChrIA|EXCEPTION_ACCESS_VIOLATION_READ (2061 crashes)
     14% (294/2061) vs.   2% (3792/172261) avsda.dll (Avira AV)
     16% (331/2061) vs.   4% (7355/172261) browsemngr.dll (Babylon)
     12% (246/2061) vs.   6% (9621/172261) mgAdaptersProxy.dll (Adapter Proxy)
      6% (127/2061) vs.   1% (1351/172261) browsemngr-15.0.dll (Babylon for Fx 15)
  StrCmpNIA|EXCEPTION_ACCESS_VIOLATION_READ (169 crashes)
     18% (30/169) vs.   2% (3792/172261) avsda.dll (Avira AV)
     28% (47/169) vs.  18% (30249/172261) mdnsNSP.dll (Bonjour)
     13% (22/169) vs.   4% (7355/172261) browsemngr.dll (Babylon)
*16.0:
  StrChrIA|EXCEPTION_ACCESS_VIOLATION_READ (423 crashes)
     32% (135/423) vs.   7% (4599/65384) browsemngr.dll (Babylon)
     17% (74/423) vs.   2% (1530/65384) avsda.dll (Avira AV)
     23% (98/423) vs.  14% (9376/65384) mdnsNSP.dll (Bonjour)
     13% (57/423) vs.   6% (4109/65384) mgAdaptersProxy.dll (Adapter Proxy)
      7% (28/423) vs.   1% (825/65384) browsemngr-16.0.dll (Babylon for Fx 16)
  StrCmpNIA|EXCEPTION_ACCESS_VIOLATION_READ (66 crashes)
     27% (18/66) vs.   7% (4599/65384) browsemngr.dll (Babylon)
     14% (9/66) vs.   6% (4109/65384) mgAdaptersProxy.dll (Adapter Proxy)
     21% (14/66) vs.  14% (9376/65384) mdnsNSP.dll (Bonjour)
No volume on 17 to justify tracking (yet), please re-nominate if this shows up in any significant number on 17 Beta.
i was also running into it with a test vm - https://crash-stats.mozilla.com/report/index/bp-2d73ad55-9b52-4faa-8ac9-1be3b2121017 so let me know if you need any info
(In reply to Carsten Book [:Tomcat] from comment #61)
> i was also running into it with a test vm -
> https://crash-stats.mozilla.com/report/index/bp-2d73ad55-9b52-4faa-8ac9-
> 1be3b2121017 so let me know if you need any info
This one might be caused by Babylon.
Crash Signature: [@ StrChrIA ] [@ StrCmpNIA] → [@ StrChrIA ] [@ StrCmpNIA] [@ StrStrIA ]
It's #2 top browser crasher in 19.0.2, #3 in 20.0b4 and 21.0a2, and #6 in 22.0a1.
That higher volume is likely caused by FireJump (see http://firejump.net/) where it's correlated at 22%:
  StrChrIA|EXCEPTION_ACCESS_VIOLATION_READ (3995 crashes)
     22% (862/3995) vs.   1% (1095/139851) firejump@firejump.net (1.0.2.5)
  StrCmpNIA|EXCEPTION_ACCESS_VIOLATION_READ (406 crashes)
     21% (86/406) vs.   1% (1095/139851) firejump@firejump.net (1.0.2.5)
  StrStrIA|EXCEPTION_ACCESS_VIOLATION_READ (18 crashes)
     33% (6/18) vs.   1% (1095/139851) firejump@firejump.net (1.0.2.5)
Depends on: 849925
Is there an example on crash-stats where FireJump is installed?
Looking at the comments for this crash, it looks like most of them are in German. Is it possible that this is correlated to the German version of Firefox or some German software other than FireJump?
(In reply to Jorge Villalobos [:jorgev] from comment #66)
> Is it possible that this is correlated to the German version of Firefox or some German
> software other than FireJump?
It might be indeed a geographical correlation. It's also correlated to Avira Antivirus and BrowserProtect (known as adware) with similar proportions:
     22% (1073/4944) vs.   3% (4865/170379) avsda.dll
          5% (249/4944) vs.   1% (1469/170379) 12.3.2.15
         16% (794/4944) vs.   2% (3117/170379) 13.4.2.360
          0% (1/4944) vs.   0% (8/170379) 8.0.0.5
          0% (1/4944) vs.   0% (10/170379) 9.0.9.0
     21% (1019/4944) vs.   7% (11388/170379) BrowserProtect.dll
          0% (12/4944) vs.   0% (506/170379) 2.5.1005.80
          0% (5/4944) vs.   0% (219/170379) 2.5.986.67
          0% (4/4944) vs.   0% (522/170379) 2.6.1040.25
          0% (11/4944) vs.   0% (474/170379) 2.6.1070.41
         19% (922/4944) vs.   5% (8980/170379) 2.6.1095.52
          1% (58/4944) vs.   0% (579/170379) 2.6.1125.80
          0% (7/4944) vs.   0% (107/170379) 2.6.1184.107

Note that we don't have correlations per LSP.
We continue to see a high inflow of people on #firefox.de on moznet that are affected by this.
There is a bug about blocklisting the extension, BUT the problem seems to be caused by:
C:\Windows\System32\dhRichClient3.dll
C:\Windows\System32\sqlite36_engine.dll
which are tied additionally into windows through hundreds of registry entries.
So blocklisting would likely NOT help.
Depends on: 908263
Whiteboard: [crashkill][crashkill-outreach][crashkill-blocklist][explosive?] → [crashkill][crashkill-outreach][crashkill-blocklist][explosive?][unactionable]
Keywords: topcrashtopcrash-win
[@ StrChrIA] seems to be spiking a bit recently on Release:
https://crash-analysis.mozilla.com/rkaiser/2013-11-12/2013-11-12.firefox.25.explosiveness.html

Currently at #9 in 7-day and 3-day Release top-crash reports.
StrChrIA is currently the #11 crash on v26 release, and it also has some lesser-hitting friends, StrCmpNIA and StrStrIA.

StrStrIA calls both of the other two functions. In Windows 8.1, the calls got inlined, so we only see StrStrIA on the stack. In older Windows we see one of the other callees, with StrStrIA below it.

v26 has user-agent locale logging, which confirms the previous observations that almost all affected users are running in German.

All the crashes are running off the end of a memory page, for example trying to do a 2-byte read at address 12345FFF, where 12345FFF is valid memory but the next byte 12346000 is decommitted.

Unwinding stacks by hand, I see frames like this:
StrStrIA
XXXX1A50
XXXX91C1
_PR_MD_RECV
...where those XXXX are two frames of dynamically-allocated code (allocation flags PAGE_EXECUTE_READWRITE, current flags PAGE_EXECUTE_READ, size 0xA000) always with the same two offsets. The return address at _PR_MD_RECV indicates that it was trying to call the |recv| API.

Here's the difficulty: Other than the German language, I don't see any major correlations among recent crash reports. Yes, some users have avsda, but not enough to explain all the crashes. I don't see any suspicious extensions, modules, or LSPs across sufficiently many reports. These dynamic code pages must be coming from some external software that the minidumps don't show.
Combining the three known signatures, this is the #1 crash for users running with locale=de, with 9.5% of their crashes. https://crash-stats.mozilla.com/search/?useragent_locale=de&_facets=signature
dmajor, if we had a user who was experiencing this could we figure out the cause by logging stacks for the VirtualAlloc of the executable code pages? If we have people coming to #firefox.de with this, there are probably some technical users among them.
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #72)
> dmajor, if we had a user who was experiencing this could we figure out the
> cause by logging stacks for the VirtualAlloc of the executable code pages?

Yes, that could likely point to the source of the allocation. I've put instructions here: https://wiki.mozilla.org/Tracing_VirtualAlloc_With_Xperf
Henrik, any chance you can investigate this?
Flags: needinfo?(hskupin)
Sorry, but I really don't have the time for anything outside of automation at the moment. There should also be other people who have a better knowledge of debugging on Windows.
Flags: needinfo?(hskupin)
Updating the rank info since it's been quite some time.

@StrChrIA is still around in significant volume:
* Firefox 29: #36 @ 0.33% (+0.05%) w/37 crashes
* Firefox 28: #23 @ 0.56% (-0.06%) w/59 crashes
* Firefox 27: #34 @ 0.29% (+0.06%) w/667 crashes
* Firefox 26: #11 @ 1.08% (+0.07%) w/6632 crashes

@StrCmpNIA appears to have disappeared.
@StrStrIA appears to have all but disappeared with 11 crashes on Firefox 28 and none on any other version.
We contacted several affected users and the commonality between them is a program called Gutscheinfilter or G-Filter, from gutscheinfilter.de.

The G-Filter installer writes two exe files in System32 and registers them as Windows services. The first is usually called gfiltersvc.exe though we have also seen gfiltersvc0.exe. The second file has a random name, chosen by taking the name of a nearby dll in System32 and mutating one character. The random file can be identified by its timestamp which is very close to that of gfiltersvc.exe. Also, the two services can be identified by their blank description in the Windows services manager.

The crashes happened while the software was using a network API hook to overwrite some security-related HTTP headers. Given that behavior and the shifting filenames, this software does not seem trustworthy.

Some versions of G-Filter offer an uninstaller. On others, stopping the two services and deleting their executables appears to be sufficient to remove the software.

A few weeks ago this was easily the top crash in German-speaking locales, but it has now completely fallen off the charts. Perhaps antiviruses have started blocking this software, or perhaps there is a new version that doesn't crash.
This reminds me, when we find a piece of software that is obviously malware, we have in the past submitted it to anti-virus companies so they add it to their detection. In such cases, please get a hold of the offending DLL/EXE and contact me personally.
> A few weeks ago this was easily the top crash in German-speaking locales, but it has now completely 
> fallen off the charts. Perhaps antiviruses have started blocking this software, or perhaps there is a > new version that doesn't crash. 
 
I noted >1000 Win7 >1000 Fx35.0.1 crashes and #59 top crash I believe.

Trying to assist user in Sumo thread https://support.mozilla.org/en-US/questions/1046938 Would be grateful for any assistance either in the thread or comments here. 

bp-84f387ee-c576-449a-b00c-d0ccc2150215 

Crashing Thread
0 	shlwapi.dll 	StrChrIA 	
1 	shlwapi.dll 	StrStrIA 	
2 	ws2_32.dll 	WSARecv 	
3 	wsock32.dll 	recv 	
4 	nss3.dll 	_PR_MD_RECV 	nsprpub/pr/src/md/windows/w95sock.c
5 	nss3.dll 	SocketRead 	nsprpub/pr/src/io/prsocket.c
6 	xul.dll 	nsSocketInputStream::Read(char*, unsigned int, unsigned int*) 	netwerk/base/src/nsSocketTransport2.cpp
7 	xul.dll 	mozilla::net::nsHttpConnection::OnWriteSegment(char*, unsigned int, unsigned int*) 	netwerk/protocol/http/nsHttpConnection.cpp
8 	xul.dll 	mozilla::net::nsHttpTransaction::WritePipeSegment(nsIOutputStream*, void*, char*, unsigned int, unsigned int, unsigned int*) 	netwerk/protocol/http/nsHttpTransaction.cpp
9 	xul.dll 	nsPipeOutputStream::WriteSegments(tag_nsresult (*)(nsIOutputStream*, void*, char*, unsigned int, unsigned int, unsigned int*), void*, unsigned int, unsigned int*) 	xpcom/io/nsPipe3.cpp
10 	xul.dll 	mozilla::net::nsHttpTransaction::WriteSegments(mozilla::net::nsAHttpSegmentWriter*, unsigned int, unsigned int*) 	netwerk/protocol/http/nsHttpTransaction.cpp
11 	xul.dll 	mozilla::net::nsHttpConnection::OnSocketReadable() 	netwerk/protocol/http/nsHttpConnection.cpp
12 	xul.dll 	mozilla::net::nsHttpConnection::OnInputStreamReady(nsIAsyncInputStream*) 	netwerk/protocol/http/nsHttpConnection.cpp
13 	xul.dll 	nsSocketInputStream::OnSocketReady(tag_nsresult) 	netwerk/base/src/nsSocketTransport2.cpp
14 	xul.dll 	nsSocketTransportService::DoPollIteration(bool) 	netwerk/base/src/nsSocketTransportService2.cpp
15 	xul.dll 	nsSocketTransportService::Run() 	netwerk/base/src/nsSocketTransportService2.cpp
16 	xul.dll 	nsThread::ProcessNextEvent(bool, bool*) 	xpcom/threads/nsThread.cpp
17 	xul.dll 	nsComponentManagerImpl::CreateInstance(nsID const&, nsISupports*, nsID const&, void**) 	xpcom/components/nsComponentManager.cpp
18 	xul.dll 	MessageLoop::DoWork() 	ipc/chromium/src/base/message_loop.cc
19 	xul.dll 	MessageLoop::RunHandler() 	ipc/chromium/src/base/message_loop.cc
20 	xul.dll 	nsThread::ThreadFunc(void*) 	xpcom/threads/nsThread.cpp
21 	msvcr100.dll 	msvcr100.dll@0x8b581 	
22 	ntdll.dll 	LdrpGetShimEngineInterface 	
23 	ntdll.dll 	_RtlUserThreadStart
(In reply to John Hesling [:John99] from comment #80)
> I noted >1000 Win7 >1000 Fx35.0.1 crashes and #59 top crash I believe.
> 
> Trying to assist user in Sumo thread
> https://support.mozilla.org/en-US/questions/1046938 Would be grateful for
> any assistance either in the thread or comments here. 

I don't think anyone has made any progress in resolving this issue, since this bug report hasn't really been updated in almost a year. If the information in comment 78 does not help then I'm not sure what else to suggest.
Whiteboard: [crashkill][crashkill-outreach][crashkill-blocklist][explosive?][unactionable] → [crashkill][crashkill-outreach][crashkill-blocklist][explosive?][unactionable][necko-backlog]
Bulk change to priority: https://bugzilla.mozilla.org/show_bug.cgi?id=1399258
Priority: -- → P1
Bulk change to priority: https://bugzilla.mozilla.org/show_bug.cgi?id=1399258
Priority: P1 → P3

Bug cannot be a "critical" and a P3.

Severity: critical → normal

Putting this back in the triage queue and restoring critical flag.

Severity: normal → critical
Priority: P3 → --

Will look through the reports for LSPs or other stuff.

Assignee: nobody → honzab.moz
Priority: -- → P2
Whiteboard: [crashkill][crashkill-outreach][crashkill-blocklist][explosive?][unactionable][necko-backlog] → [crashkill][crashkill-outreach][crashkill-blocklist][explosive?][unactionable][necko-triaged]
You need to log in before you can comment on or make changes to this bug.