Closed Bug 725945 Opened 12 years ago Closed 12 years ago

Crash in nsCacheService::Unlock

Categories

(Core :: Networking: Cache, defect)

Product:
Core
Component:
Networking: Cache
Platform:
x86
Windows XP
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla13
Tracking Flags:
Tracking Status
firefox11 --- verified
firefox12 --- verified
firefox-esr10 - wontfix

People

(Reporter: scoobidiver, Unassigned)

References

Details

(Keywords: crash, topcrash)

Crash Data

It's a new crash that first appeared in 13.0a1/20120204 and in 12.0a2/20120205.

The m-c regression window is:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=e777c939a3f9&tochange=766a59650976
The Aurora regression window is:
http://hg.mozilla.org/releases/mozilla-aurora/pushloghtml?fromchange=7d6a23398033&tochange=63278b4cbe87
It might be caused by http://hg.mozilla.org/releases/mozilla-aurora/rev/63278b4cbe87 which is a backout of bug 715308 and bug 721510.

Signature 	_chkstk More Reports Search
UUID	b6522c64-9b99-4ef5-a39a-5cee02120210
Date Processed	2012-02-10 07:56:24
Uptime	1909
Last Crash	32.0 minutes before submission
Install Age	16.2 hours since version was first installed.
Install Time	2012-02-09 15:46:45
Product	Firefox
Version	13.0a1
Build ID	20120209031242
Release Channel	nightly
OS	Windows NT
OS Version	5.1.2600 Service Pack 3
Build Architecture	x86
Build Architecture Info	GenuineIntel family 15 model 4 stepping 1
Crash Reason	EXCEPTION_ACCESS_VIOLATION_READ
Crash Address	0x165fd000
App Notes 	
AdapterVendorID: 0x1002, AdapterDeviceID: 0x94c3, AdapterSubsysID: 00000000, AdapterDriverVersion: 8.930.0.0
D3D10 Layers? D3D10 Layers-
D3D9 Layers? D3D9 Layers+
EMCheckCompatibility	True

Frame 	Module 	Signature [Expand] 	Source
0 	xul.dll 	_chkstk 	chkstk.asm:99
1 	xul.dll 	nsCacheService::Unlock 	netwerk/cache/nsCacheService.cpp:2337
2 	xul.dll 	nsCacheEntryDescriptor::nsOutputStreamWrapper::LazyInit 	netwerk/cache/nsCacheEntryDescriptor.cpp:781
3 	xul.dll 	nsCacheEntryDescriptor::nsOutputStreamWrapper::Write 	netwerk/cache/nsCacheEntryDescriptor.cpp:839
4 	xul.dll 	nsStreamCopierIB::ConsumeInputBuffer 	xpcom/io/nsStreamUtils.cpp:519
5 	xul.dll 	nsPipeInputStream::ReadSegments 	xpcom/io/nsPipe3.cpp:799
6 	nspr4.dll 	nspr4.dll@0x2a0f 	
7 	xul.dll 	nsStreamCopierIB::DoCopy 	xpcom/io/nsStreamUtils.cpp:536
8 	xul.dll 	nsAStreamCopier::Process 	xpcom/io/nsStreamUtils.cpp:319
9 	nspr4.dll 	_MD_CURRENT_THREAD 	nsprpub/pr/src/md/windows/w95thred.c:308
10 	nspr4.dll 	PR_ExitMonitor 	nsprpub/pr/src/threads/prmon.c:132
11 	xul.dll 	nsAStreamCopier::Run 	xpcom/io/nsStreamUtils.cpp:435
12 	xul.dll 	nsThreadPool::Run 	xpcom/threads/nsThreadPool.cpp:217
13 	xul.dll 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:657
14 	ntdll.dll 	RtlpAdjustHeapLookasideDepth 	
15 	nspr4.dll 	_PR_NativeRunThread 	nsprpub/pr/src/threads/combined/pruthr.c:426
16 	nspr4.dll 	pr_root 	nsprpub/pr/src/md/windows/w95thred.c:122
17 	msvcr100.dll 	_callthreadstartex 	f:\\dd\\vctools\\crt_bld\\self_x86\\crt\\src\\threadex.c:314
18 	msvcr100.dll 	msvcr100.dll@0x8b581 	
19 	msvcr100.dll 	_threadstartex 	f:\\dd\\vctools\\crt_bld\\self_x86\\crt\\src\\threadex.c:292
20 	kernel32.dll 	BaseThreadStart 

More reports at:
https://crash-stats.mozilla.com/report/list?signature=_chkstk
Why do you think this was caused by the backout of bug 715308 and bug 721510?
(In reply to Justin Lebar [:jlebar] from comment #1)
> Why do you think this was caused by the backout of bug 715308 and bug 721510?
http://hg.mozilla.org/releases/mozilla-aurora/rev/63278b4cbe87 is the only changeset that belongs to the two suspected (low volume crash) regression ranges.
I'd be surprised if this backout was causing new crashes.  It restored Aurora to old code.

Jason, did we mess with the cache code recently?
Nothing seems to have been checked into netwerk/cache since Jan 28th
It's #17 top crasher in 11.0b2.

Before Fx12, there's also this stack:
Frame 	Module 	Signature [Expand] 	Source
0 	xul.dll 	_chkstk 	chkstk.asm:99
1 	xul.dll 	nsCacheService::Unlock 	netwerk/cache/nsCacheService.cpp:2302
2 	xul.dll 	nsCacheServiceAutoLock::~nsCacheServiceAutoLock 	netwerk/cache/nsCacheService.h:323
3 	xul.dll 	nsCacheEntryDescriptor::nsInputStreamWrapper::LazyInit 	netwerk/cache/nsCacheEntryDescriptor.cpp:579
4 	xul.dll 	nsCacheEntryDescriptor::nsInputStreamWrapper::Read 	netwerk/cache/nsCacheEntryDescriptor.cpp:604
5 	xul.dll 	nsInputStreamTransport::Read 	netwerk/base/src/nsStreamTransportService.cpp:232
6 	xul.dll 	nsStreamCopierOB::FillOutputBuffer 	xpcom/io/nsStreamUtils.cpp:562
7 	xul.dll 	nsPipeOutputStream::WriteSegments 	xpcom/io/nsPipe3.cpp:1137
8 	xul.dll 	nsStreamCopierOB::DoCopy 	xpcom/io/nsStreamUtils.cpp:579
9 	xul.dll 	nsAStreamCopier::Process 	xpcom/io/nsStreamUtils.cpp:319
10 	nspr4.dll 	PR_ExitMonitor 	nsprpub/pr/src/threads/prmon.c:132
No longer depends on: 715308, 721510
Keywords: regression
Version: 12 Branch → unspecified
Keywords: topcrash
This is a stack-handling bug which has nothing especially to do with the cache. The stack MSVC gives me is:

 	xul.dll!_chkstk()  Line 99	Asm
>	xul.dll!nsCacheService::Unlock()  Line 2339	C++
 	xul.dll!nsCacheEntryDescriptor::nsOutputStreamWrapper::LazyInit()  Line 781	C++
 	xul.dll!nsCacheEntryDescriptor::nsOutputStreamWrapper::Write(buf=0x17448000, count=0x00001000, result=0x165ffe2c)  Line 840	C++
 	xul.dll!nsStreamCopierIB::ConsumeInputBuffer(inStr=0x139d3748, closure=0x165ffe54, buffer=0x17448000, offset=0x00000000, count=0x00001000, countWritten=0x165ffe2c)  Line 520	C++
 	xul.dll!nsPipeInputStream::ReadSegments(writer=0x0105678f, closure=0x165ffe54, count=0x00008000, readCount=0x165ffe5c)  Line 801	C++
 	xul.dll!nsStreamCopierIB::DoCopy(sourceCondition=0x165ffe84, sinkCondition=0x165ffe88)  Line 536	C++
 	xul.dll!nsAStreamCopier::Process()  Line 321	C++
 	xul.dll!nsAStreamCopier::Run()  Line 438	C++
 	xul.dll!nsThreadPool::Run()  Line 219	C++
 	xul.dll!nsThread::ProcessNextEvent(mayWait=true, result=0x165fff4c)  Line 658	C++
 	xul.dll!nsThread::ThreadFunc(arg=0x10bc0a01)  Line 289	C++
 	nspr4.dll!_PR_NativeRunThread(arg=0x060bb040)  Line 448	C
 	nspr4.dll!pr_root(arg=0x060bb040)  Line 122	C
 	msvcr100.dll!_callthreadstartex()  Line 314	C
 	msvcr100.dll!_threadstartex(ptd=0x0037f5b0)  Line 292	C

But what's actually happening is that nsCacheService::Unlock is calling into nsTArray_base<nsTArrayDefaultAllocator>::SwapArrayElements<nsTArrayDefaultAllocator> via inlinng and that function is allocating 8k on the stack.

In nsThread::ThreadFunc $ESP	0x165fff24	unsigned long
In nsCacheService::Unlock 		$ESP	0x165ffd84	unsigned long

In the minidump we get memory "up" to 0x165FF638, which would be... less than 2k of stack, which seems highly improbable.

So we're really not using that much stack, so we really shouldn't be running out of stack here. I would blame bug 687722 but that doesn't fit the regression range at all... scoobidivier, is it possible that this is a new manifestation of an older signature, something perhaps related to nsTArray? I'd also blame the MSVC10 switch for the new signature except that doesn't affect 12.0a2.

jlebar, thoughts?
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #6)
> I would blame bug 687722 but that doesn't fit the regression range at all
The _chkstk is a meta crash signature so the regression range is hard to determine because stacks are various. In addition, the regression range found in 12 doesn't explain the regression in 11.

> scoobidivier, is it possible that this is a new manifestation of an older 
> signature, something perhaps related to nsTArray?
Crash signatures containing nsCacheService::Unlock and nsTArray have a very low volume:
https://crash-stats.mozilla.com/query/query?product=Firefox&version=ALL%3AALL&range_value=4&range_unit=weeks&date=02%2F16%2F2012+17%3A51%3A26&query_search=signature&query_type=contains&query=nsCacheService%3A%3AUnlock&reason=&build_id=&process_type=any&hang_type=any&do_query=1
The 10.0.1 crash signature is _alloca_probe, which is #15 top browser crash in 10.0.1.
97% of _alloca_probe crashes are on Windows XP.
67% of _chkstk crashes are on Windows XP.
Crash Signature: [@ _chkstk] → [@ _chkstk] [@ _alloca_probe]
OS: Windows 7 → Windows XP
Summary: Crash in nsCacheService::Unlock @ _chkstk → Crash in nsCacheService::Unlock
We should probably Skiplist both of those signatures.
Depends on: 727963
There is a spike in crashes for the _alloca_probe crash signature from 13.0a1/20120215. The regression range for the spike is:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=60edf587f4af&tochange=d45c7d7b0079
The _chkstk crash signature is currently #1 top crasher in 10.0.2, #12 in 11.0b2 and #14 in 12.0a2.
tracking-firefox10: --- → ?
tracking-firefox11: --- → ?
tracking-firefox12: --- → ?
(In reply to Ted Mielczarek [:ted, :luser] from comment #9)
> We should probably Skiplist both of those signatures.

I'll look into that, but given the spike, I'd think that this won't give us much diversity.


According to the Signature Summary on https://crash-stats.mozilla.com/report/list?signature=_chkstk most of those crashes are on Windows XP.
And https://crash-stats.mozilla.com/report/list?signature=_alloca_probe shows a very similar picture. Does that help us any?
re: comment 10; did the _alloca_probe signature spike while the _chkstk signature remained constant, or did it just replace it? Can we look backwards and see if there is an obvious regression range for either signature in the firefox 10 nightly cycle?
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #13)
> re: comment 10; did the _alloca_probe signature spike while the _chkstk
> signature remained constant, or did it just replace it?

The different signatures are spiking on different versions, _chkstk on 10, _alloca_probe on 11.


Here are module correlations calculated from yesterday's data for both signatures and the main versions they appear in:


Correlations for Firefox 10.0.2 Windows NT - Signature: _chkstk

93% (2386/2562) vs.  22% (11753/52727) cscui.dll
93% (2384/2562) vs.  22% (11751/52727) cscdll.dll
75% (1914/2562) vs.   7% (3912/52727) wuapi.dll
67% (1712/2562) vs.   8% (4427/52727) cabinet.dll
98% (2507/2562) vs.  45% (23738/52727) cryptui.dll
89% (2268/2562) vs.  37% (19540/52727) samlib.dll
62% (1584/2562) vs.  12% (6435/52727) tapi32.dll
60% (1526/2562) vs.  10% (5468/52727) msv1_0.dll
99% (2535/2562) vs.  51% (27007/52727) wldap32.dll
99% (2529/2562) vs.  51% (27054/52727) lz32.dll
99% (2533/2562) vs.  52% (27275/52727) xpsp2res.dll
94% (2404/2562) vs.  47% (24990/52727) msacm32.drv
94% (2407/2562) vs.  48% (25091/52727) midimap.dll
94% (2409/2562) vs.  48% (25199/52727) wdmaud.drv
94% (2406/2562) vs.  48% (25321/52727) msacm32.dll
54% (1382/2562) vs.   8% (4346/52727) browseui.dll
58% (1488/2562) vs.  13% (6808/52727) atl.dll
54% (1375/2562) vs.   9% (4557/52727) cryptdll.dll
99% (2541/2562) vs.  55% (28868/52727) iphlpapi.dll
99% (2536/2562) vs.  55% (28865/52727) hnetcfg.dll
52% (1324/2562) vs.   8% (4360/52727) sensapi.dll
99% (2541/2562) vs.  56% (29293/52727) wshtcpip.dll
67% (1713/2562) vs.  24% (12444/52727) rtutils.dll
99% (2534/2562) vs.  56% (29544/52727) comres.dll
62% (1586/2562) vs.  19% (10257/52727) rasapi32.dll
62% (1589/2562) vs.  20% (10411/52727) rasman.dll
99% (2537/2562) vs.  57% (30244/52727) ws2help.dll
98% (2498/2562) vs.  57% (30299/52727) urlmon.dll
98% (2515/2562) vs.  58% (30434/52727) netapi32.dll
97% (2477/2562) vs.  58% (30409/52727) apphelp.dll
99% (2533/2562) vs.  63% (33105/52727) imagehlp.dll
99% (2527/2562) vs.  63% (33093/52727) mpr.dll
99% (2534/2562) vs.  69% (36526/52727) secur32.dll
100% (2550/2562) vs.  72% (37792/52727) t2embed.dll
87% (2225/2562) vs.  61% (32041/52727) icm32.dll
99% (2527/2562) vs.  73% (38408/52727) wininet.dll
35% (906/2562) vs.  10% (5460/52727) mlang.dll
57% (1467/2562) vs.  32% (17133/52727) MSCTF.dll
77% (1977/2562) vs.  53% (27730/52727) normaliz.dll
83% (2129/2562) vs.  61% (31902/52727) ntmarta.dll
99% (2529/2562) vs.  77% (40418/52727) shdocvw.dll
100% (2556/2562) vs.  79% (41733/52727) nssckbi.dll
100% (2556/2562) vs.  80% (41926/52727) nssdbm3.dll
100% (2556/2562) vs.  80% (41930/52727) freebl3.dll
41% (1055/2562) vs.  22% (11445/52727) MSCTFIME.IME
38% (979/2562) vs.  19% (10180/52727) msctfime.ime
77% (1985/2562) vs.  59% (30929/52727) iertutil.dll
96% (2470/2562) vs.  78% (41290/52727) rsaenh.dll
22% (559/2562) vs.   6% (3290/52727) msxml3.dll
28% (713/2562) vs.  12% (6518/52727) winsta.dll
18% (459/2562) vs.   2% (1311/52727) netrap.dll
18% (461/2562) vs.   3% (1325/52727) netui1.dll
18% (461/2562) vs.   3% (1325/52727) netui0.dll
52% (1342/2562) vs.  37% (19483/52727) ntshrui.dll
100% (2551/2562) vs.  85% (44982/52727) rasadhlp.dll
99% (2544/2562) vs.  85% (44922/52727) feclient.dll
16% (412/2562) vs.   2% (1111/52727) msgina.dll
32% (829/2562) vs.  18% (9696/52727) d3d8thk.dll
32% (829/2562) vs.  18% (9697/52727) d3d9.dll
100% (2556/2562) vs.  86% (45410/52727) winrnr.dll
16% (412/2562) vs.   3% (1321/52727) odbcint.dll
16% (411/2562) vs.   3% (1319/52727) odbc32.dll
16% (416/2562) vs.   3% (1724/52727) GrooveSystemServices.dll
14% (364/2562) vs.   2% (982/52727) sti.dll
18% (464/2562) vs.   6% (3087/52727) davclnt.dll
18% (463/2562) vs.   6% (3090/52727) drprov.dll
18% (462/2562) vs.   6% (3077/52727) ntlanman.dll
17% (425/2562) vs.   5% (2713/52727) wzcsapi.dll
100% (2552/2562) vs.  89% (46680/52727) dnsapi.dll
20% (519/2562) vs.   9% (5009/52727) ieframe.dll
100% (2552/2562) vs.  89% (46939/52727) mscms.dll
19% (491/2562) vs.   9% (4668/52727) GrooveShellExtensions.dll
19% (491/2562) vs.   9% (4668/52727) GrooveUtil.DLL
19% (491/2562) vs.   9% (4668/52727) GrooveNew.DLL
19% (494/2562) vs.   9% (4829/52727) ATL80.dll
12% (298/2562) vs.   2% (1012/52727) pdfshell.dll
100% (2551/2562) vs.  90% (47555/52727) wintrust.dll
100% (2550/2562) vs.  90% (47639/52727) userenv.dll
11% (289/2562) vs.   2% (1195/52727) shimgvw.dll
100% (2562/2562) vs.  92% (48265/52727) browsercomps.dll
17% (435/2562) vs.   9% (4906/52727) msi.dll
13% (336/2562) vs.   5% (2877/52727) activeds.dll
13% (332/2562) vs.   5% (2845/52727) mprapi.dll
13% (338/2562) vs.   6% (3041/52727) adsldpc.dll
10% (257/2562) vs.   3% (1530/52727) credui.dll
10% (256/2562) vs.   3% (1538/52727) netshell.dll
9% (222/2562) vs.   2% (913/52727) GrooveMisc.dll
18% (468/2562) vs.  11% (5982/52727) GdiPlus.dll
9% (219/2562) vs.   2% (1102/52727) dot3dlg.dll
9% (219/2562) vs.   2% (1104/52727) dot3api.dll
8% (196/2562) vs.   1% (699/52727) MSOHEVI.DLL
9% (220/2562) vs.   2% (1196/52727) eappcfg.dll
9% (220/2562) vs.   2% (1196/52727) onex.dll
9% (220/2562) vs.   2% (1196/52727) eappprxy.dll
7% (187/2562) vs.   1% (355/52727) atl90.dll
100% (2562/2562) vs.  94% (49605/52727) softokn3.dll
100% (2562/2562) vs.  94% (49632/52727) firefox.exe
100% (2556/2562) vs.  94% (49510/52727) crypt32.dll
100% (2556/2562) vs.  94% (49520/52727) msasn1.dll
100% (2562/2562) vs.  94% (49646/52727) xpcom.dll
11% (283/2562) vs.   5% (2771/52727) msvcp60.dll
100% (2562/2562) vs.  94% (49758/52727) dbghelp.dll
10% (244/2562) vs.   4% (2364/52727) PortableDeviceApi.dll
10% (255/2562) vs.   5% (2562/52727) wlanapi.dll


Correlations for Firefox 11.0 Windows NT - Signature: _alloca_probe

     90% (371/411) vs.  20% (4323/21205) cscui.dll
     90% (371/411) vs.  20% (4329/21205) cscdll.dll
     99% (407/411) vs.  31% (6561/21205) cryptui.dll
     99% (407/411) vs.  34% (7286/21205) lz32.dll
     99% (408/411) vs.  35% (7494/21205) wldap32.dll
     99% (408/411) vs.  37% (7802/21205) xpsp2res.dll
     99% (408/411) vs.  38% (8139/21205) iphlpapi.dll
     99% (408/411) vs.  43% (9030/21205) hnetcfg.dll
     60% (248/411) vs.   4% (849/21205) wuapi.dll
     99% (408/411) vs.  43% (9127/21205) wshtcpip.dll
     81% (333/411) vs.  26% (5488/21205) samlib.dll
     99% (408/411) vs.  45% (9469/21205) mpr.dll
     56% (232/411) vs.   6% (1229/21205) browseui.dll
     58% (238/411) vs.   8% (1604/21205) msv1_0.dll
     99% (408/411) vs.  49% (10482/21205) comres.dll
     59% (241/411) vs.   9% (1952/21205) atl.dll
     99% (408/411) vs.  50% (10642/21205) ws2help.dll
     59% (241/411) vs.  10% (2049/21205) tapi32.dll
    100% (410/411) vs.  52% (11109/21205) t2embed.dll
     99% (408/411) vs.  53% (11262/21205) netapi32.dll
    100% (410/411) vs.  55% (11755/21205) imagehlp.dll
     50% (205/411) vs.   6% (1268/21205) sensapi.dll
     94% (386/411) vs.  51% (10746/21205) apphelp.dll
     62% (253/411) vs.  20% (4199/21205) rtutils.dll
    100% (410/411) vs.  59% (12509/21205) shdocvw.dll
     46% (190/411) vs.   6% (1201/21205) cryptdll.dll
     59% (241/411) vs.  18% (3837/21205) rasapi32.dll
     59% (241/411) vs.  18% (3889/21205) rasman.dll
     44% (181/411) vs.   5% (1047/21205) cabinet.dll
    100% (410/411) vs.  61% (13027/21205) nssckbi.dll
     96% (395/411) vs.  58% (12280/21205) msacm32.drv
     96% (396/411) vs.  58% (12338/21205) msacm32.dll
     96% (396/411) vs.  58% (12343/21205) wdmaud.drv
    100% (410/411) vs.  62% (13164/21205) nssdbm3.dll
    100% (410/411) vs.  62% (13165/21205) freebl3.dll
     95% (392/411) vs.  58% (12282/21205) midimap.dll
    100% (410/411) vs.  65% (13717/21205) feclient.dll
    100% (411/411) vs.  67% (14171/21205) winrnr.dll
    100% (411/411) vs.  69% (14724/21205) browsercomps.dll
    100% (411/411) vs.  70% (14869/21205) softokn3.dll
    100% (411/411) vs.  70% (14871/21205) firefox.exe
    100% (411/411) vs.  70% (14902/21205) xpcom.dll
     98% (404/411) vs.  69% (14542/21205) secur32.dll
     96% (393/411) vs.  66% (14021/21205) urlmon.dll
    100% (410/411) vs.  71% (15068/21205) dbghelp.dll
     59% (242/411) vs.  30% (6403/21205) MSCTF.dll
     50% (204/411) vs.  22% (4564/21205) gkmedias.dll
     97% (398/411) vs.  69% (14580/21205) rsaenh.dll
    100% (411/411) vs.  72% (15256/21205) rasadhlp.dll
     45% (186/411) vs.  20% (4203/21205) MSCTFIME.IME
    100% (411/411) vs.  75% (15855/21205) dnsapi.dll
     73% (300/411) vs.  49% (10351/21205) ntmarta.dll
     56% (229/411) vs.  33% (7036/21205) ntshrui.dll
     99% (408/411) vs.  77% (16304/21205) wininet.dll
     28% (114/411) vs.   6% (1190/21205) msxml3.dll
     31% (129/411) vs.  10% (2079/21205) winsta.dll
     74% (303/411) vs.  54% (11389/21205) normaliz.dll
     23% (96/411) vs.   4% (807/21205) GrooveSystemServices.dll
     29% (120/411) vs.  11% (2269/21205) ATL80.dll
     29% (119/411) vs.  11% (2237/21205) GrooveShellExtensions.dll
     29% (119/411) vs.  11% (2238/21205) GrooveNew.DLL
     29% (119/411) vs.  11% (2238/21205) GrooveUtil.DLL
     20% (81/411) vs.   2% (375/21205) netrap.dll
     20% (81/411) vs.   2% (377/21205) netui0.dll
     20% (81/411) vs.   2% (377/21205) netui1.dll
     86% (354/411) vs.  69% (14579/21205) icm32.dll
     19% (78/411) vs.   2% (346/21205) msgina.dll
     30% (122/411) vs.  13% (2730/21205) msctfime.ime
     19% (78/411) vs.   2% (524/21205) odbc32.dll
     19% (78/411) vs.   2% (525/21205) odbcint.dll
     20% (83/411) vs.   4% (848/21205) IDMShellExt.dll
    100% (411/411) vs.  84% (17812/21205) wintrust.dll
     18% (72/411) vs.   2% (338/21205) shimgvw.dll
     20% (82/411) vs.   4% (948/21205) davclnt.dll
     20% (82/411) vs.   4% (950/21205) drprov.dll
     20% (81/411) vs.   4% (944/21205) ntlanman.dll
     16% (67/411) vs.   1% (262/21205) sti.dll
     22% (92/411) vs.   8% (1668/21205) ieframe.dll
    100% (411/411) vs.  86% (18198/21205) mswsock.dll
     15% (60/411) vs.   1% (302/21205) pdfshell.dll
     23% (93/411) vs.  10% (2112/21205) GdiPlus.dll
     74% (303/411) vs.  61% (12948/21205) iertutil.dll
    100% (411/411) vs.  88% (18694/21205) setupapi.dll
     13% (52/411) vs.   2% (419/21205) GrooveMisc.dll
     14% (59/411) vs.   4% (792/21205) PortableDeviceApi.dll
     21% (85/411) vs.  10% (2158/21205) idmmkb.dll
     10% (41/411) vs.   0% (97/21205) atl90.dll
    100% (410/411) vs.  91% (19249/21205) mscms.dll
     19% (79/411) vs.  11% (2265/21205) msvcr71.dll
    100% (410/411) vs.  92% (19538/21205) userenv.dll
      9% (39/411) vs.   3% (583/21205) wzcsapi.dll
      8% (31/411) vs.   1% (175/21205) MSOHEVI.DLL
     11% (47/411) vs.   5% (1054/21205) GrooveIntlResource.dll
      7% (27/411) vs.   1% (160/21205) pshook.dll
    100% (411/411) vs.  94% (19988/21205) crypt32.dll
    100% (411/411) vs.  94% (19988/21205) msasn1.dll
     29% (121/411) vs.  24% (5039/21205) linkinfo.dll
     12% (48/411) vs.   6% (1352/21205) RadioWMPCoreGecko11.dll
      8% (32/411) vs.   3% (552/21205) netshell.dll
      8% (32/411) vs.   3% (559/21205) credui.dll
     15% (62/411) vs.  10% (2130/21205) wtsapi32.dll
tracking-firefox10: + → ?
tracking-firefox11: + → ?
tracking-firefox12: + → ?
> So we're really not using that much stack, so we really shouldn't be running out of stack 
> here.

The 8k nsAutoTArray in nsTArray::SwapArrayElements is likely excessive.  I bet we could dial this down to 512 bytes with no ill effects.

Would you like to try this and see if it helps?
Here are today's correlations without Windows and Firefox DLLs:
* 10.0.2:
  _chkstk|EXCEPTION_ACCESS_VIOLATION_READ (2768 crashes)
     17% (477/2768) vs.   4% (2016/56926) GrooveSystemServices.dll (MS Office Groove)
     20% (563/2768) vs.   9% (5189/56926) GrooveShellExtensions.dll (MS Office Groove)
     20% (563/2768) vs.   9% (5191/56926) GrooveUtil.DLL (MS Office Groove)
     20% (563/2768) vs.   9% (5191/56926) GrooveNew.DLL (MS Office Groove)
     13% (354/2768) vs.   2% (1193/56926) pdfshell.dll (Adobe PDF Shell Extension)
      9% (257/2768) vs.   2% (1055/56926) GrooveMisc.dll (MS Office Groove)
      8% (209/2768) vs.   1% (780/56926) MSOHEVI.DLL (MS Office Groove)

* 11.0 Beta:
  _alloca_probe|EXCEPTION_ACCESS_VIOLATION_READ (771 crashes)
     23% (178/771) vs.   5% (1426/29707) GrooveSystemServices.dll (MS Office Groove)
     27% (205/771) vs.  12% (3504/29707) GrooveShellExtensions.dll (MS Office Groove)
     27% (205/771) vs.  12% (3505/29707) GrooveNew.DLL (MS Office Groove)
     27% (205/771) vs.  12% (3505/29707) GrooveUtil.DLL (MS Office Groove)
     17% (128/771) vs.   4% (1199/29707) IDMShellExt.dll (Internet Download Manager)
     14% (110/771) vs.   2% (544/29707) pdfshell.dll (Adobe PDF Shell Extension)
      12% (92/771) vs.   3% (801/29707) GrooveMisc.dll (MS Office Groove)
     18% (139/771) vs.  10% (3088/29707) idmmkb.dll (Internet Download Manager)
      6% (46/771) vs.   1% (231/29707) pshook.dll (Punto Switcher)
     11% (84/771) vs.   6% (1702/29707) GrooveIntlResource.dll (MS Office Groove)
BTW, _chkstk also spikes somewhat on 9, and _alloca_probe also spikes on 12. _chkstk isn't around in any relevant volume on 11, but did rise somewhat on 12 as well. On 13, both have risen somewhat.
Crash Signature: [@ _chkstk] [@ _alloca_probe] → [@ _chkstk] [@ _chkstk | nsCacheService::Unlock()] [@ _alloca_probe] [@ _alloca_probe | nsCacheService::Unlock()]
Justin, we have no clue where to proceed with this one. If this is a low risk change, can we try it on a beta and see if it has an impact?
(In reply to Justin Lebar [:jlebar] from comment #15)
> Would you like to try this and see if it helps?

If this is low risk, I'm all for trying and seeing if it gives us a lead.


Also, correlations tell us that cscui.dll / cscdll.dll are loaded in only 20% of overall crashes but way over 90% of those with the signatures tracked here. This is some Windows-internal caching service, right? Does this give us a lead? Something in the system we access in those cases but not always in the (other) usual cases?
Okay, let's discuss taking the change in beta in bug 729453.
(In reply to Justin Lebar [:jlebar] from comment #20)
> Okay, let's discuss taking the change in beta in bug 729453.
It has been refused. So what's the plan here for Beta?
This landed on trunk, can we verify whether it completely or partially fixed the crash volume?
I requested re-triage based on this.
By no side effects, I meant no depending crashes.
Crash Signature: [@ _chkstk] [@ _chkstk | nsCacheService::Unlock()] [@ _alloca_probe] [@ _alloca_probe | nsCacheService::Unlock()] → [@ _chkstk] [@ _chkstk | nsCacheService::Unlock() ] [@ _alloca_probe] [@ _alloca_probe | nsCacheService::Unlock() ]
So it looks as though the patch in bug 729453 has eliminated this issue. I don't find any of these these crashes in 11b5 yet. Probably worth keeping this open just a bit longer so we can be sure it doesn't appear in some other form.
Still seems to be good. Removing tracking flag.
tracking-firefox10: + → ---
tracking-firefox11: + → ---
tracking-firefox12: + → ---
There are no crashes in 11.0 and above.
There are 12 crashes in 10.0.3esr making it #3 top browser crasher.
Status: NEW → RESOLVED
Closed: 12 years ago
status-firefox-esr10: --- → affected
tracking-firefox-esr10: --- → ?
Resolution: --- → FIXED
Target Milestone: --- → mozilla13
Marking verified 11 & 12 based on comment 28 and having checked crashstats.
status-firefox11: fixed → verified
status-firefox12: fixed → verified
[Triage Comment]
With only 200K users on ESR and relatively low crash numbers comparatively, this isn't really a valid 'topcrash' and so I'm removing tracking.

If orgs come back to use about these issues, we will look at fixing.
tracking-firefox-esr10: ? → -
(In reply to Lukas Blakk [:lsblakk] from comment #30)
> With only 200K users on ESR
You should remove the throttling on ESR like it's done in Fennec. I filed bug 741710.
I'm pretty sure there's no throttling on ESR.
It's #5 top browser crasher (#3 without Flash 11.3 crashes on the browser side) on 10.0.5 ESR.

top 5 crashers are enough to qualify it for an uplift (see bug 700493, bug 729401, and bug 673543).
tracking-firefox-esr10: - → ?
Tracking bug 729453 instead, since that fixes this top ESR crasher.
tracking-firefox-esr10: ? → -
You need to log in before you can comment on or make changes to this bug.