818962 - crash in nsHttpConnectionMgr::ProcessPendingQForEntry

Status: VERIFIED FIXED

(Core :: Networking: HTTP, defect)

Description

[Scoobidiver (away)]

It started spiking in today's nightly and is #1 top crasher in this build. The regression range is:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=1942b4d64dc8&tochange=3c81e1c0d252
It's likely a regression from bug 792438.

Signature 	memmove | nsTArray_base<nsTArrayDefaultAllocator>::ShiftData(unsigned int, unsigned int, unsigned int, unsigned int, unsigned int) | nsTArray<nsXPCWrappedJS*, nsTArrayDefaultAllocator>::RemoveElementsAt(unsigned int, unsigned int) | nsHttpConnectionMgr... More Reports Search
UUID	35ba4fac-3d0e-4ca5-9334-3338f2121206
Date Processed	2012-12-06 18:01:34
Uptime	58
Last Crash	1.2 hours before submission
Install Age	2.2 hours since version was first installed.
Install Time	2012-12-06 15:52:13
Product	Firefox
Version	20.0a1
Build ID	20121206030737
Release Channel	nightly
OS	Windows NT
OS Version	6.1.7601 Service Pack 1
Build Architecture	x86
Build Architecture Info	GenuineIntel family 6 model 30 stepping 5
Crash Reason	EXCEPTION_ACCESS_VIOLATION_WRITE
Crash Address	0x67035000
App Notes 	
AdapterVendorID: 0x1002, AdapterDeviceID: 0x68b8, AdapterSubsysID: 25431002, AdapterDriverVersion: 8.753.0.0
DWrite? DWrite+ 
EMCheckCompatibility	False
Adapter Vendor ID	0x1002
Adapter Device ID	0x68b8
Total Virtual Memory	4294836224
Available Virtual Memory	3106492416
System Memory Use Percentage	26
Available Page File	14091247616
Available Physical Memory	6292250624

Frame 	Module 	Signature 	Source
0 	msvcr100.dll 	memmove 	f:\dd\vctools\crt_bld\SELF_X86\crt\src\Intel\MEMCPY.ASM:185
1 	xul.dll 	nsTArray_base<nsTArrayDefaultAllocator>::ShiftData 	obj-firefox/dist/include/nsTArray-inl.h:245
2 	xul.dll 	nsTArray<nsXPCWrappedJS*,nsTArrayDefaultAllocator>::RemoveElementsAt 	obj-firefox/dist/include/nsTArray.h:946
3 	xul.dll 	nsHttpConnectionMgr::ProcessPendingQForEntry 	netwerk/protocol/http/nsHttpConnectionMgr.cpp:957
4 	xul.dll 	nsHttpConnectionMgr::ProcessAllTransactionsCB 	netwerk/protocol/http/nsHttpConnectionMgr.cpp:747
5 	xul.dll 	nsBaseHashtable<nsCStringHashKey,nsCOMPtr<mozIStorageAsyncStatement>,mozIStorage 	obj-firefox/dist/include/nsBaseHashtable.h:419
6 	xul.dll 	PL_DHashTableEnumerate 	obj-firefox/xpcom/build/pldhash.cpp:717
7 	xul.dll 	nsBaseHashtable<nsStringHashKey,nsRefPtr<gfxFontFamily>,gfxFontFamily*>::Enumera 	obj-firefox/dist/include/nsBaseHashtable.h:223
8 	xul.dll 	nsHttpConnectionMgr::OnMsgProcessPendingQ 	netwerk/protocol/http/nsHttpConnectionMgr.cpp:2048
9 	xul.dll 	nsHttpConnectionMgr::nsConnEvent::Run 	netwerk/protocol/http/nsHttpConnectionMgr.h:545
10 	xul.dll 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:627
11 	xul.dll 	nsSocketTransportService::Run 	netwerk/base/src/nsSocketTransportService2.cpp:653
12 	xul.dll 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:627
13 	xul.dll 	nsThread::ThreadFunc 	xpcom/threads/nsThread.cpp:265
14 	nspr4.dll 	_PR_NativeRunThread 	nsprpub/pr/src/threads/combined/pruthr.c:395
15 	nspr4.dll 	pr_root 	nsprpub/pr/src/md/windows/w95thred.c:90
16 	msvcr100.dll 	_callthreadstartex 	f:\dd\vctools\crt_bld\self_x86\crt\src\threadex.c:314
17 	msvcr100.dll 	_threadstartex 	f:\dd\vctools\crt_bld\self_x86\crt\src\threadex.c:292
18 	kernel32.dll 	BaseThreadInitThunk 	
19 	ntdll.dll 	__RtlUserThreadStart 	
20 	ntdll.dll 	_RtlUserThreadStart 	

More reports at:
https://crash-stats.mozilla.com/report/list?signature=memmove+|+nsTArray_base%3CnsTArrayDefaultAllocator%3E%3A%3AShiftData%28unsigned+int%2C+unsigned+int%2C+unsigned+int%2C+unsigned+int%2C+unsigned+int%29+|+nsTArray%3CnsXPCWrappedJS*%2C+nsTArrayDefaultAllocator%3E%3A%3ARemoveElementsAt%28unsigned+int%2C+unsigned+int%29+|+nsHttpConnectionMgr...
https://crash-stats.mozilla.com/report/list?signature=memcpy+|+nsTArray%3Cmozilla%3A%3Awidget%3A%3AGfxInfoCollectorBase*%2C+nsTArrayDefaultAllocator%3E%3A%3ARemoveElementsAt%28unsigned+int%2C+unsigned+int%29+|+nsHttpConnectionMgr%3A%3AProcessPendingQForEntry%28nsHttpConnectionMgr%3A%3AnsConnectionEntry*%2C+bool%29

Comment 1

[Patrick McManus [:mcmanus]]

always a good feeling to be the top crasher.

I'm on it.

Comment 2

[Patrick McManus [:mcmanus]]

neither of the "more reports at" links show me anything.

I use

https://crash-stats.mozilla.com/query/query?product=Firefox&version=Firefox%3A20.0a1&range_value=1&range_unit=weeks&date=12%2F06%2F2012+18%3A55%3A55&query_search=signature&query_type=contains&query=RemoveElementsAt&reason=&build_id=&process_type=any&hang_type=any&do_query=1

just ~20 stacks can make you the top crasher.. wow.

Comment 3

[Patrick McManus [:mcmanus]]

Attachment: patch 0

This patch deals with any stack on TryToDispatchTransaction() that modifies mPendingQ .. that is what appears to be going on from the stack trace although exactly how is a bit of a mystery.

Comment 4

[Honza Bambas (:mayhemer)]

Comment on attachment 689406 [details] [diff] [review]
patch 0

Review of attachment 689406 [details] [diff] [review]:
-----------------------------------------------------------------

Hope there is no other hidden problem.  Looks good.

r=honzab.

Comment 5

[Patrick McManus [:mcmanus]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/a850f2aafdb1

Comment 6

[Scoobidiver (away)]

(In reply to Patrick McManus [:mcmanus] from comment #2)
> just ~20 stacks can make you the top crasher.. wow.
Currently there are 230 crashes in today's build (follow the links in comment 0 - add ... in the location bar at the end of the first link) with 165 ADUs so there are about 140 crashes/100 ADU while a normal nightly build has about 15 overall crashes/100 ADU. So yes it's a top crasher.

Comment 7

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/a850f2aafdb1

Comment 8

[Paul Silaghi, QA [:pauly]]

No crashes on FF 20 checking the crashstats. Verified fixed.