Closed Bug 598774 Opened 9 years ago Closed 8 years ago

Crash @ fastcopy_I | _VEC_memcpy | _VEC_memcpy | nsVoidArray::operator=

Categories

(Core :: General, defect, critical)

7 Branch
x86
Windows 7
defect
Not set
critical

Tracking

()

RESOLVED WORKSFORME
Tracking Status
firefox7 + affected

People

(Reporter: scoobidiver, Unassigned)

References

Details

(Keywords: crash, regression)

Crash Data

Build : Mozilla/5.0 (Windows NT 6.1; rv:2.0b6) Gecko/20100101 Firefox/4.0b6

This is a crash signature that is present in 3.6.9, 4.0b5 and 4.0b6.
It is #38 top crasher for the last 2 weeks in 4.0b6.

Here are the comments about this crash signature in 4.0b5 and b6 :
"Every auto launch of Firefox beta crashes. I have to manually restart it in order to get it to work"
"Anytime i try to sign into google documents, it will crash"
"When i started this happened always with mozilla firefox v4"
"What I have noticed is the CPU usage always 100%, as soon as I start Firefox Beta 5 ... this is not good"

Signature	fastcopy_I
UUID	a3094374-bb6c-48b4-9aa2-258652100922
Time 	2010-09-22 10:28:31.556488
Uptime	2352
Install Age	419386 seconds (4.9 days) since version was first installed.
Product	Firefox
Version	4.0b6
Build ID	20100914083612
Branch	2.0
OS	Windows NT
OS Version	6.1.7600
CPU	x86
CPU Info	GenuineIntel family 6 model 23 stepping 10
Crash Reason	EXCEPTION_ACCESS_VIOLATION_READ
Crash Address	0xc846000
App Notes 	AdapterVendorID: 10de, AdapterDeviceID: 06e4

Crashing Thread
Frame 	Module 	Signature [Expand] 	Source
0 	mozcrt19.dll 	fastcopy_I 	
1 	mozcrt19.dll 	_VEC_memcpy 	
2 	xul.dll 	nsXULTooltipListener::`scalar deleting destructor' 	
3 	mozcrt19.dll 	_VEC_memcpy 	
4 	xul.dll 	nsTArray_base::ShiftData 	obj-firefox/xpcom/build/nsTArray.cpp:175
5 	xul.dll 	nsEventListenerManager::Disconnect 	content/events/src/nsEventListenerManager.cpp:1234
6 	xul.dll 	nsEventListenerManager::cycleCollection::Unlink 	content/events/src/nsEventListenerManager.cpp:348
7 	xul.dll 	nsCycleCollector::CollectWhite 	xpcom/base/nsCycleCollector.cpp:1784
8 	xul.dll 	nsCycleCollector::FinishCollection 	xpcom/base/nsCycleCollector.cpp:2655
9 	xul.dll 	nsCycleCollector::Collect 	xpcom/base/nsCycleCollector.cpp:2471
Crash Signature: [@ fastcopy_I ]
This signature still garners a fair amount of crashes per week: https://crash-stats.mozilla.com/report/list?signature=fastcopy_I

The user in Bug 656612 seems to hit the crash pretty frequently.
Depends on: 656612
It's #55 top crasher in 5.0, #94 in 6.0 and #11 in 7.0a2.

Stack traces are different from the ones in comment 0 and look like:
Frame 	Module 	Signature [Expand] 	Source
0 	mozcrt19.dll 	fastcopy_I 	
1 	mozcrt19.dll 	_VEC_memcpy 	
2 	mozcrt19.dll 	_VEC_memcpy 	
3 	xul.dll 	nsVoidArray::operator= 	obj-firefox/xpcom/build/nsVoidArray.cpp:339
4 	xul.dll 	nsCOMArray_base::Clear 	obj-firefox/xpcom/build/nsCOMArray.cpp:175
5 	xul.dll 	nsRefPtr<nsPresContext>::~nsRefPtr<nsPresContext> 	obj-firefox/dist/include/nsAutoPtr.h:969
6 	xul.dll 	nsDocument::~nsDocument 	content/base/src/nsDocument.cpp:1681
7 	xul.dll 	nsHTMLDocument::`vector deleting destructor' 	
8 	xul.dll 	nsNodeUtils::LastRelease 	content/base/src/nsNodeUtils.cpp:326
9 	xul.dll 	nsXMLDocument::Release 	content/xml/document/src/nsXMLDocument.cpp:249
10 	xul.dll 	DoDeferredRelease<nsISupports*> 	js/src/xpconnect/src/xpcjsruntime.cpp:607
11 	xul.dll 	XPCJSRuntime::GCCallback 	js/src/xpconnect/src/xpcjsruntime.cpp:917
...

5.0 crash stats show a correlation with the RealPlayer Browser Record Plugin extension:
65% (75/116) vs.   2% (893/47599) {ABDE892B-13A8-4d1b-88E6-365A6E755758}
      65% (75/116) vs.   1% (257/47599) 1.0.0
78% (90/116) vs.   1% (285/47599) nprndlffbrowserrecordext.dll
      69% (80/116) vs.   1% (248/47599) 1.0.0.333
       9% (10/116) vs.   0% (37/47599) 1.0.0.334
Summary: Firefox 4.0b6 crash [@ fastcopy_I ] → Crash [@ fastcopy_I ] mainly with the RealPlayer Browser Record Plugin extension
(In reply to Scoobidiver from comment #2)
> 5.0 crash stats show a correlation with the RealPlayer Browser Record Plugin
> extension:
> 65% (75/116) vs.   2% (893/47599) {ABDE892B-13A8-4d1b-88E6-365A6E755758}


Can we do something about that? Should we contact them? Block it? Something else?
It's #6 top browser crasher in 7.0b1.
It is #93 top crasher in 6.0 and #11 in 7.0b1.
There's no longer a correlation with RealPlayer in 6.0 or 7.0.
Summary: Crash [@ fastcopy_I ] mainly with the RealPlayer Browser Record Plugin extension → Crash [@ fastcopy_I ]
not something that the release drivers are going to track.
we were averaging about 750 crashes per day for this durning july. now averaging around 920 over the last 30 days.

distribution is across many releases but it does look like a volume regression some where along the path in the recent releases.  firefox 7 has almost as many crashes per day as firefox 6 with substantially fewer users.

         fastcopy_I
date     total    breakdown by build
         crashes  count build, count build, ...

20110823 754  	161 6.02011081116, 
        		153 7.02011081615, 	114 3.6.202011080313, 
        		54 5.02011061515, 	40 5.0.12011070718, 
        		27 3.6.182011061423, 	23 4.0.12011041322, 
        		19 3.6.32010040108, 	14 4.02011031805, 
        		12 5.02011042714, 	10 3.6.132010120307, 
        		9 8.0a22011082204, 	8 3.62010011514, 
        		7 4.0b102011012116, 	6 4.0b72010110414, 
        		6 3.6.152011030302, 	5 5.02011052709, 
        		5 3.6.172011042014, 	5 3.6.162011031913, 
        		4 4.0b32010080519, 	4 3.6.42010051314, 
        		3 9.0a12011082211, 	3 6.02011080403, 
        		3 6.02011071317, 	3 4.0b82010121417, 
        		3 3.6.82010072215, 	3 3.6.122010102621, 
        		3 3.5.22009072922, 	3 3.5.192011042014, 
        		3 3.0.192010031422, 	2 8.0a22011082104, 
        		2 7.0a22011081604, 	2 6.02011072115, 
        		2 5.02011061417, 	2 4.0b22010072019, 
        		2 4.0b122011022221, 	2 4.0b112011020314, 
        		2 3.6.92010082415, 	2 3.6.62010062523, 
        		2 3.6.42010061114, 	2 3.6.112010101211, 
        		2 3.6.102010091412, 	1 9.0a12011082303, 
        		1 9.0a12011082203, 	1 9.0a12011082103, 
        		1 8.0a22011082004, 	1 7.0a22011081004, 
        		1 5.02011051719, 	1 4.0b62010091408, 
        		1 4.0b12010063014, 	1 3.6b52009120414, 
        		1 3.6.72010070102, 	1 3.6.3plugin12010040510, 
        		1 3.6.22010031607, 	1 3.6.192011070719, 
        		1 3.6.142011020713, 	1 3.52009061517, 
        		1 3.0.162009120208, 	1 3.0.12008070208,
Something definitely regressed here. I looked at some data on 7.0 beta compared to 6.0 beta. We are getting about 105 to 200 of these on release versions ie: 5.0, 6.0. for 7.0 beta we have over 200 a day. During 6.0 beta we had about 50 a day.
It's #3 top browser crasher (w/o hangs) in 7.0b2.
Ok, so how do we reason about this? It's coming down to the wire for 7
The crash ratio between 7.0 (about 3000 crashes/week/1M ADU) and 6.0.2 (23 crashes/week/1M ADU) is about 130.

According to crash stats in Firefox 7 Aurora:
https://crash-stats.mozilla.com/report/list?signature=fastcopy_I&&range_value=4&range_unit=weeks&version=Firefox:7.0a2
the regression range for the spike is:
http://hg.mozilla.org/releases/mozilla-aurora/pushloghtml?fromchange=bcd26d45856a&tochange=d80597c09388
bug 678588 changed singleDeviceInterval from 24 hours to 1 hour. It has only highlighted an existing problem.
Blocks: 678588
Version: Trunk → 7 Branch
I backed out bug 678588 on mozilla-beta8 but kept it everywhere else for now.
(In reply to Christian Legnitto [:LegNeato] from comment #12)
> I backed out bug 678588 on mozilla-beta8 but kept it everywhere else for now.
s/mozilla-beta8/mozilla-beta7/
Indeed, sorry about that...was looking at something 8 related :-/ scoobi, you are a rockstar
I've been looking through crash reports and the stack traces for fastcopy are all over the place (because fastcopy is part of the implementation for malloc, realloc, etc on Win32). I have a nagging suspicion we are dealing with a heap corruption, but it isn't obvious where it is occurring (as with most heap corruptions).

I'm thinking analysis of the stack straces for common frames would be helpful. The folks in #metrics and #breakpad say that chofmann has a tool to do this.

chofmann: can you explain what your stack analysis tool does and help us use it, if appropriate?
We have some other hints pointing at memory corruption introduced sometime this summer; jduell was tracking one last night responsible for ~13% of crashes on fennec that looks like mem corruption, and I've heard a few other anecdotes about hard-to-pin-down crashes pointing there.
I'm currently working on parsing *all* of the crash reports from the last 30 days (from https://crash-analysis.mozilla.com/crash_analysis/) to see if I can find any common patterns, particularly with regards to the stack frames and version numbers. I'll update the bug when I have data.
Here is the breakdown of counts of crashes in fastcopy_I grouped by version for the past 30 days:

3.0     12
3.0.1   9
3.0.10  11
3.0.11  4
3.0.13  1
3.0.14  1
3.0.15  3
3.0.16  2
3.0.17  1
3.0.19  147
3.0.3   14
3.0.4   4
3.0.5   4
3.0.6   5
3.0.7   5
3.0.8   7
3.1b3   12
3.5     44
3.5.1   1
3.5.10  3
3.5.11  5
3.5.13  2
3.5.14  1
3.5.15  4
3.5.16  9
3.5.17  2
3.5.18  6
3.5.19  36
3.5.2   71
3.5.3   12
3.5.4   4
3.5.5   26
3.5.6   11
3.5.7   6
3.5.8   3
3.5.9   9
3.5b4   2
3.6     312
3.6.10  152
3.6.11  61
3.6.12  152
3.6.13  476
3.6.14  38
3.6.15  116
3.6.16  110
3.6.17  217
3.6.18  562
3.6.19  34
3.6.2   78
3.6.20  2990
3.6.21  2179
3.6.22  3235
3.6.3   488
3.6.3plugin1    6
3.6.4   220
3.6.6   140
3.6.7   68
3.6.8   178
3.6.9   45
3.6b1   1
3.6b2   1
3.6b3   2
3.6b4   2
3.6b5   16
3.7a3   3
3.7a5   2
3.7a6pre        1
4.0     257
4.0.1   685
4.0b1   73
4.0b10  52
4.0b11  55
4.0b12  56
4.0b12pre       1
4.0b13pre       2
4.0b2   117
4.0b3   119
4.0b4   34
4.0b5   62
4.0b6   39
4.0b7   115
4.0b7pre        2
4.0b8   78
4.0b9   44
4.2a1pre        1
5.0     1722
5.0.1   540
5.0a2   1
6.0     3181
6.0.1   1705
6.0.2   3519
6.0a2   1
7.0     6389
7.0a2   56
8.0a1   15
8.0a2   438
9.0a1   108

Given the number of crashes in the released versions (up to 6.x at the time this was written), it appears we are dealing with an outstanding bug. But, the counts in the unreleased versions are troubling given the percentage of users on those versions. 7, 8, and 9 have about the same number of crashes (7006) as 6.x (8405).

Anyway, I don't think there is much new data in this comment that hasn't already been said.

Next up, I will be looking at the stack traces for all the crashes (31851 total) looking for frame correlations.
First, I've created some tools to enable easier crash analysis. Code available at https://github.com/mozilla-services/socorro-crash-utils. I've tried to include as much as possible in a Python module, which others should just be able to pick up and import to grasp the power.

Anyway, I've fetched ~20k crash dumps so far. The 100 most common frames *anywhere in the crashed stack* are:

28114   _VEC_memcpy
24249
19960   fastcopy_I
5716    js_Interpret
5502    nsJSContext::EvaluateString(nsAString_internal const &,void *,nsIPrincipal *,char const *,unsigned int,unsigned int,nsAString_internal *,int *)
5491    nsScriptLoader::ProcessRequest(nsScriptLoadRequest *)
5477    nsScriptLoader::EvaluateScript(nsScriptLoadRequest *,nsString const &)
5405    JS_EvaluateUCScriptForPrincipals
5370    js_Execute
5350    PushBackTrackState
5321    str_match
5311    nsScriptLoader::ProcessPendingRequests()
5299    js_ExecuteRegExp
5220    DoMatch
5200    nsCOMArray_base::RemoveObject(nsISupports *)
3911    nsThread::ProcessNextEvent(int,int *)
3641    MessageLoop::RunHandler()
3450    wmain
3430    nsCOMPtr_base::~nsCOMPtr_base()
3390    nsTimerImpl::Fire()
3389    MessageLoop::Run()
3382    XRE_main
3378    mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate *)
3270    JS_GC
3261    nsVoidArray::operator=(nsVoidArray const &)
3221    nsBaseAppShell::Run()
3217    nsCOMArray_base::Clear()
3169    nsTimerEvent::Run()
3040    nsAppStartup::Run()
3014    _initterm
3014    _RTC_Initialize
2988    js_GC(JSContext *,JSCompartment *,JSGCInvocationKind)
2983    XPCJSRuntime::GCCallback(JSContext *,JSGCStatus)
2982    DoDeferredRelease<nsISupports *>
2981    DOMGCCallback
2962    JS_CompartmentGC
2943    nsXPConnect::Collect()
2917    nsNodeUtils::LastRelease(nsINode *)
2897    nsDocument::~nsDocument()
2770    BaseProcessStart
2696    nsXPConnect::GarbageCollect()
2692    nsJSContext::GarbageCollectNow()
2194    Str
2018    _MD_CURRENT_THREAD
1886    nsHTMLDocument::Release()
1760    nsHTMLDocument::`scalar deleting destructor'(unsigned int)
1547    arena_ralloc
1476    F_1799079006_______________________________________________
1414    GetCodePageFileInfo
1371    realloc
1304    _RtlUserThreadStart
1294    JO
1066    mozilla::storage::AsyncExecuteStatements::AsyncExecuteStatements(nsTArray<mozilla::storage::StatementData,nsTArrayDefaultAllocator> &,mozilla::storage::Connection *,mozIStorageStatementCallback *)
1042    nsHTMLDocument::`vector deleting destructor'(unsigned int)
935     F_2118623609_________________________________________
884     nsXMLDocument::Release()
877     F_1229453757___________________________________________________________
876     PixelCopy32
734     JS_CallFunctionValue
714     arena_ralloc_large_grow
694     F_876307109____________________________
692     F2052336212_____________________________________________________________
682     mozilla::`anonymous namespace'::ContainerState::ProcessDisplayItems(nsDisplayList const &,mozilla::FrameLayerBuilder::Clip &)
678     F_958494385______________________
676     JA
643     F_655827159____________________________________
598     nsGenericElement::DestroyContent()
583     FindAtomW
554     arena_dalloc
551     js::ExternalInvoke(JSContext *,js::Value const &,js::Value const &,unsigned int,js::Value *,js::Value *)
520     F790412612___________________________
502     nsDOMEvent::Release()
499     InterlockedExchange
496     arena_dalloc_small
495     LdrpGetShimEngineInterface
488     nsXPCWrappedJSClass::CallMethod(nsXPCWrappedJS *,unsigned short,XPTMethodDescriptor const *,nsXPTCMiniVariant *)
488     nsXPCWrappedJS::CallMethod(unsigned short,XPTMethodDescriptor const *,nsXPTCMiniVariant *)
487     SharedStub
483     js::RunScript(JSContext *,JSScript *,js::StackFrame *)
473     PrepareAndDispatch
459     js_Invoke
437     __tmainCRTStartup
428     ScopedXPCOMStartup::~ScopedXPCOMStartup()
426     LdrpAppendToForwarderList
425     F_973733276______________________________
421     nsEventListenerManager::Disconnect()
418     nsCycleCollector::CollectWhite()
408     js::mjit::JaegerShot(JSContext *)
406     nsCycleCollector::FinishCollection()
402     F_1860184299______________________________
397     F_1572877831_________________________________
396     js::Invoke(JSContext *,js::CallArgs const &,js::ConstructOption)
395     F_1946984377_____________________
394     AssignExpr
337     js::mjit::EnterMethodJIT(JSContext *,js::StackFrame *,void *,js::Value *)
329     nsGlobalWindow::RunTimeout(nsTimeout *)
320     nsDOMEvent::`scalar deleting destructor'(unsigned int)
320     malloc
317     js::Interpret(JSContext *,js::StackFrame *,unsigned int,js::InterpMode)
316     nsIFrame::InvalidateInternal(nsRect const &,int,int,nsIFrame *,unsigned int)
OK, I finished downloading all the full crash files. Of the 31851 crashes with "fastcopy_I" in their signature in the past 30 days, here are some statistics.

Top 100 symbols occurring in any frame:

44640   _VEC_memcpy
37263
31850   fastcopy_I
8653    js_Interpret
8104    nsJSContext::EvaluateString(nsAString_internal const &,void *,nsIPrincipal *,char const *,unsigned int,unsigned int,nsAString_internal *,int *)
8091    nsScriptLoader::ProcessRequest(nsScriptLoadRequest *)
8064    nsScriptLoader::EvaluateScript(nsScriptLoadRequest *,nsString const &)
8012    PushBackTrackState
7958    JS_EvaluateUCScriptForPrincipals
7917    js_ExecuteRegExp
7902    js_Execute
7837    nsScriptLoader::ProcessPendingRequests()
7829    DoMatch
7828    str_match
7659    nsCOMArray_base::RemoveObject(nsISupports *)
6490    nsThread::ProcessNextEvent(int,int *)
6033    MessageLoop::RunHandler()
5632    nsTimerImpl::Fire()
5631    MessageLoop::Run()
5607    mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate *)
5449    nsVoidArray::operator=(nsVoidArray const &)
5386    nsCOMArray_base::Clear()
5383    wmain
5355    JS_GC
5333    nsBaseAppShell::Run()
5294    XRE_main
5256    nsTimerEvent::Run()
4976    js_GC(JSContext *,JSCompartment *,JSGCInvocationKind)
4971    XPCJSRuntime::GCCallback(JSContext *,JSGCStatus)
4969    DoDeferredRelease<nsISupports *>
4968    DOMGCCallback
4939    JS_CompartmentGC
4905    nsXPConnect::Collect()
4898    nsNodeUtils::LastRelease(nsINode *)
4846    nsDocument::~nsDocument()
4686    _initterm
4686    _RTC_Initialize
4671    nsAppStartup::Run()
4636    BaseProcessStart
4478    nsXPConnect::GarbageCollect()
4473    nsJSContext::GarbageCollectNow()
4420    nsCOMPtr_base::~nsCOMPtr_base()
3649    Str
3624    nsHTMLDocument::Release()
3402    _MD_CURRENT_THREAD
2531    nsHTMLDocument::`scalar deleting destructor'(unsigned int)
2441    arena_ralloc
2421    F_1799079006_______________________________________________
2339    GetCodePageFileInfo
2210    realloc
2160    nsHTMLDocument::`vector deleting destructor'(unsigned int)
2154    JO
1757    _RtlUserThreadStart
1649    F_2118623609_________________________________________
1473    nsRefPtr<nsPresContext>::~nsRefPtr<nsPresContext>()
1400    F_1229453757___________________________________________________________
1398    PixelCopy32
1337    mozilla::storage::AsyncExecuteStatements::AsyncExecuteStatements(nsTArray<mozilla::storage::StatementData,nsTArrayDefaultAllocator> &,mozilla::storage::Connection *,mozIStorageStatementCallback *)
1155    JS_CallFunctionValue
1151    F_876307109____________________________
1148    F2052336212_____________________________________________________________
1133    F_958494385______________________
1124    arena_ralloc_large_grow
1122    nsAppShell::Run()
1113    JA
1058    F_655827159____________________________________
1039    mozilla::`anonymous namespace'::ContainerState::ProcessDisplayItems(nsDisplayList const &,mozilla::FrameLayerBuilder::Clip &)
1027    nsXMLDocument::Release()
1024    FindAtomW
929     nsGenericElement::DestroyContent()
914     arena_dalloc
890     arena_dalloc_small
876     js::ExternalInvoke(JSContext *,js::Value const &,js::Value const &,unsigned int,js::Value *,js::Value *)
836     InterlockedExchange
825     nsXPCWrappedJS::CallMethod(unsigned short,XPTMethodDescriptor const *,nsXPTCMiniVariant *)
822     nsXPCWrappedJSClass::CallMethod(nsXPCWrappedJS *,unsigned short,XPTMethodDescriptor const *,nsXPTCMiniVariant *)
822     js_Invoke
822     SharedStub
803     PrepareAndDispatch
782     F790412612___________________________
767     js::RunScript(JSContext *,JSScript *,js::StackFrame *)
763     nsDOMEvent::Release()
713     ScopedXPCOMStartup::~ScopedXPCOMStartup()
704     F_973733276______________________________
698     __tmainCRTStartup
677     F_1860184299______________________________
665     LdrpGetShimEngineInterface
662     nsEventListenerManager::Disconnect()
652     F_1946984377_____________________
650     js::mjit::JaegerShot(JSContext *)
641     js::Invoke(JSContext *,js::CallArgs const &,js::ConstructOption)
614     F_1572877831_________________________________
610     nsCycleCollector::CollectWhite()
593     nsCycleCollector::FinishCollection()
575     AssignExpr
556     LdrpAppendToForwarderList
554     js::mjit::EnterMethodJIT(JSContext *,js::StackFrame *,void *,js::Value *)
554     js::Interpret(JSContext *,js::StackFrame *,unsigned int,js::InterpMode)
551     nsDOMEvent::`scalar deleting destructor'(unsigned int)
509     malloc

Top 50 symbols in any frame for stacks containing "malloc":

152     PresShell::Paint(nsIView *,nsIWidget *,nsRegion const &,nsIntRegion const &,int,int)
152     nsDisplayList::PaintForFrame(nsDisplayListBuilder *,nsRenderingContext *,nsIFrame *,unsigned int)
155     DispatchClientMessage
155     KiUserCallbackDispatcher
158     nsJSContext::EvaluateString(nsAString_internal const &,void *,nsIPrincipal *,char const *,unsigned int,unsigned int,nsAString_internal *,int *)
159     nsWindow::DispatchWindowEvent(nsGUIEvent *,nsEventStatus &)
160     PresShell::FlushPendingNotifications(mozFlushType)
160     nsWindow::OnPaint(HDC__ *,unsigned int)
164     nsWindow::DispatchEvent(nsGUIEvent *,nsEventStatus &)
166     nsViewManager::DispatchEvent(nsGUIEvent *,nsIView *,nsEventStatus *)
170     arena_ralloc_large_grow
173     PrepareAndDispatch
173     UserCallWinProcCheckWow
174     nsWindow::ProcessMessage(unsigned int,unsigned int &,long &,long *)
176     JS_CallFunctionValue
176     __tmainCRTStartup
177     SharedStub
177     nsXPCWrappedJS::CallMethod(unsigned short,XPTMethodDescriptor const *,nsXPTCMiniVariant *)
180     nsWindow::WindowProc(HWND__ *,unsigned int,unsigned int,long)
180     nsXPCWrappedJSClass::CallMethod(nsXPCWrappedJS *,unsigned short,XPTMethodDescriptor const *,nsXPTCMiniVariant *)
183     nsWindow::WindowProcInternal(HWND__ *,unsigned int,unsigned int,long)
184     InternalCallWinProc
185     __fnDWORD
190     _MD_CURRENT_THREAD
190     nsAppStartup::Run()
192     GetCodePageFileInfo
200     XRE_main
201     mozilla::FrameLayerBuilder::BuildContainerLayerFor(nsDisplayListBuilder *,mozilla::layers::LayerManager *,nsIFrame *,nsDisplayItem *,nsDisplayList const &)
205     wmain
214     CallWindowProcCrashProtected
223     nsBaseAppShell::Run()
234     MessageLoop::Run()
260     JA
266     nsTimerEvent::Run()
274     nsTimerImpl::Fire()
314     js_Invoke
329     arena_ralloc
364     realloc
402     js_Interpret
404     BaseProcessStart
457     mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate *)
482     JO
489     MessageLoop::RunHandler()
493     nsThread::ProcessNextEvent(int,int *)
509     malloc
722     mozilla::`anonymous namespace'::ContainerState::ProcessDisplayItems(nsDisplayList const &,mozilla::FrameLayerBuilder::Clip &)
826     Str
1161    fastcopy_I
1456    _VEC_memcpy
2863

Note how different this list is from the global one!

Top 100 {symbol, stack position} pairs (can kinda group related stacks by the frame's location in the stack):

20601   0       fastcopy_I
20426   1       _VEC_memcpy
11370   2       _VEC_memcpy
9074    0       fastcopy_I
8772    1       _VEC_memcpy
6607    3       PushBackTrackState
6533    4       js_ExecuteRegExp
6487    5       DoMatch
6439    6       str_match
6316    7       js_Interpret
6234    12      nsScriptLoader::ProcessRequest(nsScriptLoadRequest *)
6234    10      nsJSContext::EvaluateString(nsAString_internal const &,void *,nsIPrincipal *,char const *,unsigned int,unsigned int,nsAString_internal *,int *)
6233    11      nsScriptLoader::EvaluateScript(nsScriptLoadRequest *,nsString const &)
6217    9       JS_EvaluateUCScriptForPrincipals
6217    8       js_Execute
6157    13      nsCOMArray_base::RemoveObject(nsISupports *)
6126    14      nsScriptLoader::ProcessPendingRequests()
5037    3       nsVoidArray::operator=(nsVoidArray const &)
4959    4       nsCOMArray_base::Clear()
4358    6       nsDocument::~nsDocument()
3910    8       nsNodeUtils::LastRelease(nsINode *)
3257    11      XPCJSRuntime::GCCallback(JSContext *,JSGCStatus)
3256    10      DoDeferredRelease<nsISupports *>
3254    12      DOMGCCallback
3223    13      js_GC(JSContext *,JSCompartment *,JSGCInvocationKind)
3221    15      JS_GC
3219    14      JS_CompartmentGC
3205    5       nsCOMPtr_base::~nsCOMPtr_base()
3055    16      nsXPConnect::Collect()
2950    9       nsHTMLDocument::Release()
2939    21      nsThread::ProcessNextEvent(int,int *)
2927    19      nsTimerImpl::Fire()
2922    20      nsTimerEvent::Run()
2883    18      nsJSContext::GarbageCollectNow()
2883    17      nsXPConnect::GarbageCollect()
2871    24      MessageLoop::RunHandler()
2404    2       F_1799079006_______________________________________________
2104    7       nsHTMLDocument::`scalar deleting destructor'(unsigned int)
2049    0       fastcopy_I
2043    1       _VEC_memcpy
1942    31
1869    2
1858    23      mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate *)
1802    7       nsHTMLDocument::`vector deleting destructor'(unsigned int)
1734    25      MessageLoop::Run()
1723    27
1706    26      nsBaseAppShell::Run()
1691    29      XRE_main
1690    30      wmain
1688    28      nsAppStartup::Run()
1687    22      _MD_CURRENT_THREAD
1655    33      _initterm
1655    32      _RTC_Initialize
1573    37
1570    2       F_2118623609_________________________________________
1560    40
1500    36      BaseProcessStart
1489    26      MessageLoop::Run()
1339    23
1314    34
1313    25
1311    3       _VEC_memcpy
1252    28      nsBaseAppShell::Run()
1224    4       realloc
1220    2       arena_ralloc
1211    3       arena_ralloc
1201    35
1184    22      mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate *)
1121    2       arena_ralloc_large_grow
1102    3       F_876307109____________________________
1100    4       F_958494385______________________
1095    5       F2052336212_____________________________________________________________
1093    36
1082    5       nsRefPtr<nsPresContext>::~nsRefPtr<nsPresContext>()
1054    39
984     6       F_655827159____________________________________
970     33
949     3       realloc
902     2
880     30      nsAppStartup::Run()
870     32      wmain
860     9       nsXMLDocument::Release()
855     35      _initterm
855     34      _RTC_Initialize
827     31      XRE_main
804     27      mozilla::storage::AsyncExecuteStatements::AsyncExecuteStatements(nsTArray<mozilla::storage::StatementData,nsTArrayDefaultAllocator> &,mozilla::storage::Connection *,mozIStorageStatementCallback *)
796     5
784     3       js_ExecuteRegExp
777     4       F_1229453757___________________________________________________________
777     3       PixelCopy32
776     13      nsScriptLoader::ProcessPendingRequests()
774     2       PushBackTrackState
774     2       F790412612___________________________
767     4       DoMatch
764     5       str_match
758     6       js_Interpret
747     11      nsScriptLoader::ProcessRequest(nsScriptLoadRequest *)
744     8       JS_EvaluateUCScriptForPrincipals
744     7       js_Execute
742     9       nsJSContext::EvaluateString(nsAString_internal const &,void *,nsIPrincipal *,char const *,unsigned int,unsigned int,nsAString_internal *,int *)

When looking at these, keep in mind we're likely looking at multiple bugs.
I have ~80k crash dumps with fastcopy_I from the past 90 days harvested on my computer. But, I really don't know where to continue analysis beyond the primitive ones pasted above.

If anyone wants me to run a custom report, I can do that (it does take a while to fetch all the data from crash-stats). I just need to know what to run.

In hopes of reproducing the crashes, I wrote an extension that surfs the web for you, hopefully generating entropy to trigger the crash. It is kinda hacky, but works well enough: https://github.com/mozilla-services/sync-spray. Unfortunately, after running Nightly for 30 minutes on a Windows box, I was unable to get a crash. I'll keep on it and will hopefully get lucky.
Here's a breakdown of the crash counts by version and build date for the last 90 days or so:

3.0             2008-05-29 06:00:00          24
3.0             Total                        24
3.0.1           2008-07-02 08:00:00          28
3.0.1           Total                        28
3.0.10          2009-04-23 16:00:00          17
3.0.10          Total                        17
3.0.11          2009-06-02 15:00:00          11
3.0.11          Total                        11
3.0.12          2009-07-06 11:00:00           1
3.0.12          Total                         1
3.0.13          2009-07-30 22:00:00           2
3.0.13          Total                         2
3.0.14          2009-08-27 07:00:00           1
3.0.14          Total                         1
3.0.15          2009-10-16 01:00:00          10
3.0.15          Total                        10
3.0.16          2009-12-02 08:00:00           2
3.0.16          Total                         2
3.0.17          2009-12-21 16:00:00           3
3.0.17          Total                         3
3.0.18          2010-02-02 20:00:00           4
3.0.18          Total                         4
3.0.19          2010-03-14 22:00:00         413
3.0.19          Total                       413
3.0.3           2008-09-24 17:00:00          20
3.0.3           Total                        20
3.0.4           2008-10-29 20:00:00           7
3.0.4           Total                         7
3.0.5           2008-12-01 22:00:00          30
3.0.5           Total                        30
3.0.6           2009-01-19 13:00:00          11
3.0.6           Total                        11
3.0.7           2009-02-19 10:00:00          20
3.0.7           Total                        20
3.0.8           2009-03-26 09:00:00          15
3.0.8           Total                        15
3.0b4           2008-03-07 14:00:00           1
3.0b4           Total                         1
3.0b5           2008-03-26 20:00:00           1
3.0b5           Total                         1
3.1b1           2008-10-07 14:00:00           1
3.1b1           Total                         1
3.1b3           2009-03-04 23:00:00          19
3.1b3           2009-03-05 15:00:00           1
3.1b3           Total                        20
3.5             2009-06-15 17:00:00           2
3.5             2009-06-16 22:00:00           2
3.5             2009-06-24 02:00:00          97
3.5             Total                       101
3.5.1           2009-07-15 09:00:00           7
3.5.1           Total                         7
3.5.10          2010-05-04 09:00:00           7
3.5.10          Total                         7
3.5.11          2010-07-01 02:00:00          17
3.5.11          Total                        17
3.5.13          2010-09-14 13:00:00           6
3.5.13          Total                         6
3.5.14          2010-10-01 17:00:00           2
3.5.14          Total                         2
3.5.15          2010-10-26 20:00:00           9
3.5.15          Total                         9
3.5.16          2010-11-30 07:00:00          26
3.5.16          Total                        26
3.5.17          2011-01-21 15:00:00          12
3.5.17          Total                        12
3.5.18          2011-03-19 14:00:00          17
3.5.18          Total                        17
3.5.19          2011-04-20 14:00:00         110
3.5.19          Total                       110
3.5.2           2009-07-29 22:00:00         114
3.5.2           Total                       114
3.5.3           2009-08-24 10:00:00          38
3.5.3           Total                        38
3.5.4           2009-10-16 09:00:00          15
3.5.4           Total                        15
3.5.5           2009-11-02 15:00:00          57
3.5.5           Total                        57
3.5.5pre        2009-10-17 04:00:00           3
3.5.5pre        Total                         3
3.5.6           2009-12-01 22:00:00          28
3.5.6           Total                        28
3.5.7           2009-12-21 16:00:00          25
3.5.7           Total                        25
3.5.8           2010-02-02 16:00:00          11
3.5.8           Total                        11
3.5.8pre        2010-01-13 04:00:00           1
3.5.8pre        Total                         1
3.5.9           2010-03-15 08:00:00          21
3.5.9           Total                        21
3.5b4           2009-04-23 20:00:00           6
3.5b4           Total                         6
3.5b99          2009-06-05 01:00:00           1
3.5b99          Total                         1
3.6             2010-01-05 21:00:00           4
3.6             2010-01-15 14:00:00         828
3.6             Total                       832
3.6.10          2010-09-14 12:00:00         402
3.6.10          Total                       402
3.6.11          2010-09-30 13:00:00           5
3.6.11          2010-10-01 08:00:00          14
3.6.11          2010-10-12 11:00:00          83
3.6.11          Total                       102
3.6.12          2010-10-26 21:00:00         352
3.6.12          Total                       352
3.6.13          2010-11-22 05:00:00          15
3.6.13          2010-11-30 06:00:00           1
3.6.13          2010-12-03 07:00:00        1167
3.6.13          Total                      1183
3.6.14          2011-01-21 14:00:00           3
3.6.14          2011-02-07 13:00:00           2
3.6.14          2011-02-18 12:00:00          78
3.6.14          Total                        83
3.6.15          2011-03-03 02:00:00         307
3.6.15          Total                       307
3.6.16          2011-03-19 13:00:00         319
3.6.16          Total                       319
3.6.17          2011-04-14 08:00:00          31
3.6.17          2011-04-20 14:00:00         647
3.6.17          Total                       678
3.6.18          2011-06-13 02:00:00          28
3.6.18          2011-06-14 23:00:00        9148
3.6.18          Total                      9176
3.6.19          2011-07-07 19:00:00         555
3.6.19          Total                       555
3.6.19pre       2011-07-01 03:00:00           1
3.6.19pre       Total                         1
3.6.2           2010-03-16 07:00:00         262
3.6.2           Total                       262
3.6.20          2011-08-03 13:00:00        3775
3.6.20          Total                      3775
3.6.20pre       2011-08-02 03:00:00           2
3.6.20pre       Total                         2
3.6.21          2011-08-30 09:00:00        2188
3.6.21          Total                      2188
3.6.22          2011-09-02 13:00:00        3442
3.6.22          Total                      3442
3.6.23          2011-09-20 07:00:00           1
3.6.23          Total                         1
3.6.3           2010-04-01 08:00:00        1305
3.6.3           Total                      1305
3.6.3plugin1    2010-04-05 10:00:00          19
3.6.3plugin1    Total                        19
3.6.4           2010-04-13 17:00:00          20
3.6.4           2010-05-02 22:00:00           1
3.6.4           2010-05-03 12:00:00           3
3.6.4           2010-05-13 14:00:00         312
3.6.4           2010-05-23 18:00:00           7
3.6.4           2010-05-27 09:00:00          56
3.6.4           2010-06-11 14:00:00         152
3.6.4           Total                       551
3.6.5pre        2010-05-23 04:00:00           1
3.6.5pre        Total                         1
3.6.6           2010-06-25 23:00:00         332
3.6.6           Total                       332
3.6.7           2010-07-01 02:00:00          74
3.6.7           2010-07-13 13:00:00          58
3.6.7           Total                       132
3.6.8           2010-07-22 15:00:00         444
3.6.8           Total                       444
3.6.9           2010-08-24 15:00:00          96
3.6.9           Total                        96
3.6b1           2009-10-14 10:00:00           1
3.6b1           Total                         1
3.6b2           2009-11-08 18:00:00           2
3.6b2           Total                         2
3.6b3           2009-11-15 18:00:00           2
3.6b3           Total                         2
3.6b4           2009-11-24 21:00:00           7
3.6b4           Total                         7
3.6b5           2009-12-04 14:00:00          38
3.6b5           Total                        38
3.7a2           2010-02-28 16:00:00           4
3.7a2           Total                         4
3.7a3           2010-03-15 09:00:00           6
3.7a3           Total                         6
3.7a4           2010-04-07 11:00:00           1
3.7a4           Total                         1
3.7a5           2010-06-10 06:00:00           6
3.7a5           Total                         6
3.7a5pre        2010-05-11 04:00:00           1
3.7a5pre        2010-06-07 03:00:00           1
3.7a5pre        Total                         2
3.7a6pre        2010-06-18 04:00:00           3
3.7a6pre        Total                         3
4.0             2011-03-03 19:00:00          45
4.0             2011-03-18 05:00:00         932
4.0             Total                       977
4.0.1           2011-04-13 22:00:00        3484
4.0.1           Total                      3484
4.0b1           2010-06-28 21:00:00           6
4.0b1           2010-06-30 14:00:00         237
4.0b1           Total                       243
4.0b10          2011-01-21 16:00:00         158
4.0b10          Total                       158
4.0b11          2011-02-02 14:00:00           1
4.0b11          2011-02-03 14:00:00         213
4.0b11          Total                       214
4.0b12          2011-02-22 21:00:00         212
4.0b12          Total                       212
4.0b12pre       2011-02-10 03:00:00           1
4.0b12pre       2011-02-14 03:00:00           1
4.0b12pre       Total                         2
4.0b13pre       2011-03-20 03:00:00           1
4.0b13pre       2011-03-22 03:00:00           5
4.0b13pre       Total                         6
4.0b2           2010-07-20 19:00:00         209
4.0b2           Total                       209
4.0b3           2010-08-04 19:00:00           1
4.0b3           2010-08-05 19:00:00         185
4.0b3           Total                       186
4.0b4           2010-08-18 13:00:00         114
4.0b4           Total                       114
4.0b5           2010-08-31 08:00:00         116
4.0b5           Total                       116
4.0b6           2010-09-13 23:00:00           2
4.0b6           2010-09-14 08:00:00         145
4.0b6           Total                       147
4.0b7           2010-11-04 14:00:00         316
4.0b7           Total                       316
4.0b7pre        2010-10-02 04:00:00           4
4.0b7pre        Total                         4
4.0b8           2010-12-14 17:00:00         219
4.0b8           Total                       219
4.0b9           2011-01-10 19:00:00         132
4.0b9           Total                       132
4.2a1pre        2011-04-03 03:00:00           1
4.2a1pre        2011-04-11 03:00:00           1
4.2a1pre        Total                         2
5.0             2011-04-27 14:00:00         509
5.0             2011-05-17 19:00:00         322
5.0             2011-05-27 09:00:00         292
5.0             2011-06-03 10:00:00           1
5.0             2011-06-08 15:00:00         144
5.0             2011-06-13 16:00:00          36
5.0             2011-06-14 17:00:00         240
5.0             2011-06-15 15:00:00       19380
5.0             Total                     20924
5.0.1           2011-07-07 18:00:00        3463
5.0.1           Total                      3463
5.0a2           2011-04-30 04:00:00           1
5.0a2           2011-05-14 04:00:00           1
5.0a2           2011-05-18 04:00:00           1
5.0a2           2011-05-23 04:00:00           1
5.0a2           Total                         4
6.0             2011-07-05 19:00:00         434
6.0             2011-07-13 17:00:00         791
6.0             2011-07-21 15:00:00         560
6.0             2011-07-29 08:00:00         239
6.0             2011-08-04 03:00:00         499
6.0             2011-08-11 16:00:00        4473
6.0             Total                      6996
6.0.1           2011-08-30 09:00:00        1709
6.0.1           Total                      1709
6.0.2           2011-08-31 12:00:00           1
6.0.2           2011-09-02 13:00:00        3815
6.0.2           Total                      3816
6.0a1           2011-04-19 03:00:00           2
6.0a1           Total                         2
6.0a2           2011-05-27 04:00:00           8
6.0a2           2011-06-03 04:00:00           1
6.0a2           2011-06-10 04:00:00           1
6.0a2           2011-06-12 04:00:00           1
6.0a2           2011-06-19 04:00:00           1
6.0a2           2011-06-21 04:00:00           1
6.0a2           2011-06-23 04:00:00           2
6.0a2           2011-06-24 04:00:00           1
6.0a2           2011-06-25 04:00:00           3
6.0a2           2011-06-26 04:00:00           3
6.0a2           2011-06-27 04:00:00           4
6.0a2           2011-06-28 04:00:00           3
6.0a2           2011-06-29 04:00:00           1
6.0a2           2011-06-30 04:00:00           6
6.0a2           2011-07-01 04:00:00           6
6.0a2           2011-07-02 04:00:00           6
6.0a2           2011-07-03 04:00:00           4
6.0a2           2011-07-04 04:00:00           3
6.0a2           2011-07-05 04:00:00          11
6.0a2           Total                        66
7.0             2011-08-16 15:00:00        1701
7.0             2011-08-24 17:00:00        3162
7.0             2011-08-30 10:00:00         173
7.0             2011-08-31 12:00:00           1
7.0             2011-09-02 16:00:00         305
7.0             2011-09-08 13:00:00        1521
7.0             2011-09-16 09:00:00         296
7.0             Total                      7159
7.0a1           2011-06-18 03:00:00           1
7.0a1           2011-06-24 03:00:00           1
7.0a1           2011-06-25 03:00:00           4
7.0a1           2011-06-28 03:00:00           3
7.0a1           2011-06-29 03:00:00           2
7.0a1           2011-07-01 03:00:00           1
7.0a1           2011-07-02 03:00:00           1
7.0a1           2011-07-03 03:00:00           2
7.0a1           2011-07-04 03:00:00           2
7.0a1           2011-07-05 03:00:00           3
7.0a1           Total                        20
7.0a2           2011-07-06 04:00:00           9
7.0a2           2011-07-07 04:00:00          10
7.0a2           2011-07-08 04:00:00          12
7.0a2           2011-07-09 04:00:00          10
7.0a2           2011-07-10 04:00:00           6
7.0a2           2011-07-11 04:00:00          13
7.0a2           2011-07-12 04:00:00          16
7.0a2           2011-07-13 04:00:00          18
7.0a2           2011-07-14 04:00:00          10
7.0a2           2011-07-15 04:00:00          15
7.0a2           2011-07-16 04:00:00          12
7.0a2           2011-07-17 04:00:00          12
7.0a2           2011-07-18 04:00:00          15
7.0a2           2011-07-19 04:00:00          16
7.0a2           2011-07-20 04:00:00          21
7.0a2           2011-07-20 22:00:00          21
7.0a2           2011-07-22 04:00:00           3
7.0a2           2011-07-22 12:00:00           8
7.0a2           2011-07-23 04:00:00          14
7.0a2           2011-07-24 04:00:00          18
7.0a2           2011-07-25 04:00:00           9
7.0a2           2011-07-26 04:00:00          17
7.0a2           2011-07-27 04:00:00          13
7.0a2           2011-07-28 04:00:00          18
7.0a2           2011-07-29 04:00:00          14
7.0a2           2011-07-30 04:00:00          11
7.0a2           2011-07-31 04:00:00          17
7.0a2           2011-08-01 04:00:00          14
7.0a2           2011-08-02 04:00:00          11
7.0a2           2011-08-03 04:00:00          18
7.0a2           2011-08-04 04:00:00          19
7.0a2           2011-08-05 04:00:00          14
7.0a2           2011-08-06 04:00:00          20
7.0a2           2011-08-07 04:00:00          16
7.0a2           2011-08-08 04:00:00           9
7.0a2           2011-08-09 04:00:00           9
7.0a2           2011-08-10 04:00:00          24
7.0a2           2011-08-12 04:00:00          20
7.0a2           2011-08-13 04:00:00          10
7.0a2           2011-08-14 04:00:00          21
7.0a2           2011-08-15 04:00:00          15
7.0a2           2011-08-16 04:00:00         102
7.0a2           Total                       680
8.0a1           2011-07-06 03:00:00           6
8.0a1           2011-07-07 03:00:00           2
8.0a1           2011-07-08 03:00:00           7
8.0a1           2011-07-09 03:00:00           4
8.0a1           2011-07-10 03:00:00           1
8.0a1           2011-07-11 03:00:00           6
8.0a1           2011-07-12 03:00:00           7
8.0a1           2011-07-13 03:00:00           6
8.0a1           2011-07-14 03:00:00           4
8.0a1           2011-07-15 03:00:00           9
8.0a1           2011-07-16 03:00:00           4
8.0a1           2011-07-17 03:00:00           4
8.0a1           2011-07-18 03:00:00           7
8.0a1           2011-07-19 03:00:00           3
8.0a1           2011-07-20 03:00:00           4
8.0a1           2011-07-21 03:00:00           4
8.0a1           2011-07-22 03:00:00           6
8.0a1           2011-07-23 03:00:00           4
8.0a1           2011-07-24 03:00:00           2
8.0a1           2011-07-25 03:00:00           3
8.0a1           2011-07-26 03:00:00           4
8.0a1           2011-07-27 03:00:00           4
8.0a1           2011-07-27 08:00:00           5
8.0a1           2011-07-28 03:00:00           4
8.0a1           2011-07-29 03:00:00           2
8.0a1           2011-07-30 03:00:00           5
8.0a1           2011-07-31 03:00:00           4
8.0a1           2011-08-01 03:00:00           4
8.0a1           2011-08-02 03:00:00           6
8.0a1           2011-08-04 03:00:00           4
8.0a1           2011-08-05 03:00:00           2
8.0a1           2011-08-06 03:00:00           1
8.0a1           2011-08-07 03:00:00           4
8.0a1           2011-08-08 03:00:00           4
8.0a1           2011-08-09 03:00:00           5
8.0a1           2011-08-10 03:00:00           2
8.0a1           2011-08-11 03:00:00           3
8.0a1           2011-08-12 03:00:00           2
8.0a1           2011-08-13 03:00:00           6
8.0a1           2011-08-14 03:00:00           3
8.0a1           2011-08-15 03:00:00           5
8.0a1           2011-08-16 03:00:00          10
8.0a1           Total                       182
8.0a2           2011-08-17 04:00:00           1
8.0a2           2011-08-18 04:00:00           1
8.0a2           2011-08-19 04:00:00          14
8.0a2           2011-08-20 04:00:00          18
8.0a2           2011-08-21 04:00:00          15
8.0a2           2011-08-22 04:00:00          31
8.0a2           2011-08-24 04:00:00          26
8.0a2           2011-08-25 04:00:00          22
8.0a2           2011-08-26 04:00:00          14
8.0a2           2011-08-27 04:00:00          10
8.0a2           2011-08-28 04:00:00          21
8.0a2           2011-08-29 04:00:00          13
8.0a2           2011-08-30 04:00:00           9
8.0a2           2011-08-31 04:00:00          15
8.0a2           2011-09-01 04:00:00          15
8.0a2           2011-09-02 04:00:00          13
8.0a2           2011-09-03 04:00:00          14
8.0a2           2011-09-04 04:00:00          20
8.0a2           2011-09-05 04:00:00          17
8.0a2           2011-09-06 04:00:00           9
8.0a2           2011-09-07 04:00:00           9
8.0a2           2011-09-08 04:00:00          14
8.0a2           2011-09-09 04:00:00          17
8.0a2           2011-09-10 04:00:00          12
8.0a2           2011-09-11 04:00:00          11
8.0a2           2011-09-12 04:00:00           7
8.0a2           2011-09-13 04:00:00          10
8.0a2           2011-09-14 04:00:00          21
8.0a2           2011-09-15 04:00:00          24
8.0a2           2011-09-16 04:00:00          14
8.0a2           2011-09-17 04:00:00          20
8.0a2           2011-09-18 04:00:00          15
8.0a2           2011-09-19 04:00:00           8
8.0a2           2011-09-20 04:00:00           6
8.0a2           Total                       486
9.0a1           2011-08-18 03:00:00           2
9.0a1           2011-08-19 03:00:00           1
9.0a1           2011-08-20 03:00:00           4
9.0a1           2011-08-21 03:00:00           4
9.0a1           2011-08-22 03:00:00           1
9.0a1           2011-08-22 11:00:00           5
9.0a1           2011-08-23 03:00:00           4
9.0a1           2011-08-24 03:00:00           2
9.0a1           2011-08-25 03:00:00           2
9.0a1           2011-08-26 03:00:00           5
9.0a1           2011-08-27 03:00:00           5
9.0a1           2011-08-28 03:00:00           4
9.0a1           2011-08-29 05:00:00           6
9.0a1           2011-08-31 03:00:00           4
9.0a1           2011-09-01 03:00:00           5
9.0a1           2011-09-02 07:00:00           3
9.0a1           2011-09-03 03:00:00           2
9.0a1           2011-09-04 03:00:00           4
9.0a1           2011-09-05 03:00:00           1
9.0a1           2011-09-06 03:00:00           2
9.0a1           2011-09-07 03:00:00           2
9.0a1           2011-09-08 03:00:00           3
9.0a1           2011-09-09 03:00:00           1
9.0a1           2011-09-10 03:00:00           7
9.0a1           2011-09-11 03:00:00           6
9.0a1           2011-09-12 03:00:00           3
9.0a1           2011-09-13 03:00:00           2
9.0a1           2011-09-14 03:00:00           4
9.0a1           2011-09-15 03:00:00          11
9.0a1           2011-09-16 07:00:00           1
9.0a1           2011-09-17 03:00:00           4
9.0a1           2011-09-17 12:00:00           2
9.0a1           2011-09-18 03:00:00           3
9.0a1           2011-09-19 03:00:00           5
9.0a1           2011-09-20 03:00:00           1
9.0a1           2011-09-20 08:00:00           4
9.0a1           2011-09-21 03:00:00           5
9.0a1           Total                       130
(In reply to Christian Legnitto [:LegNeato] from comment #12)
> I backed out bug 678588
For an unknown reason, the backing out hasn't prevented the spike, so it is still #3 top browser crasher without hangs in 7.0b6.
After further investigations, I found that the spike happened in two close times:
* a first one at the end of 7.0a2 (see comment 11) that made it #6 top browser crasher in 7.0b1.
* a second one between 7.0b1 and 7.0b2 that made it #3 top browser crasher in 7.0b2 and above. The regression range for this second spike is: http://hg.mozilla.org/releases/mozilla-beta/pushloghtml?fromchange=00f56713eab6&tochange=4ee096ccdd5a
It scares me that bug 678422 (the nsPresArena over-allocating clownshoes one) is in that window. If we are dealing with heap corruption, it is possible those clownshoes were buffering us from a minefield of corrupted bits.
Forget about comment 24. It's because #3, #4 and #5 top browser crashers in 7.0b1 have been fixed in 7.0b2 or were startup crashes. The ratio between #1 top browser crasher and this crash signature has been pretty constant along the Beta phase, i.e. about 2.
Here are crash stats from the past 14 days:

Build     Percentage    Count  Rank
8.0a2     0.65           145    11
7.0b6     1.03           868    10
7.0b5     0.85          1268    10
6.0.2     0.11          1628   109

According to the mozilla-beta commit logs (https://hg.mozilla.org/releases/mozilla-beta/shortlog/), the backout of bug 678588 was the last commit before 7.0b6 was cut. Unfortunately, the percentage of crashes went down marginally (at least compared to the 6.0.2 0.11% baseline).

If the percentage holds up, it looks like fastcopy_I is going to be a top crasher on stable/7.0 starting tomorrow :(
I backed out bug 678588 in beta8 as well. If it is determined we don't need to do that we can reland
(In reply to Scoobidiver from comment #23)
> (In reply to Christian Legnitto [:LegNeato] from comment #12)
> > I backed out bug 678588
> For an unknown reason, the backing out hasn't prevented the spike, so it is
> still #3 top browser crasher without hangs in 7.0b6.

Actually, because of this investigation we are going to re-land bug 678588 on beta8.
(In reply to Scoobidiver from comment #23)
> For an unknown reason, the backing out hasn't prevented the spike
I think I understood. There's a delay to update before the release of a new Aurora version so every crashes in the latest Aurora 7 build were more important than in the previous ones, making think it was a spike.

7.0a1: 20 crashes/1M ADU/day
7.0a2: 130 crashes/1M ADU/day
The regression range could be in the changesets between 7.0a1/20110705 and 7.0a2/20110706:
http://hg.mozilla.org/releases/mozilla-aurora/pushloghtml?fromchange=cc6e30cce8af&tochange=92120afb1a58
No longer blocks: 678588
Keywords: topcrash
Depends on: 687672
This is pretty far down on the list for 10.0 #64 and 9.0.1 #94. I am going to take it off the top crash list.
Keywords: topcrash
Crash Signature: [@ fastcopy_I ] → [@ fastcopy_I] [@ fastcopy_I | _VEC_memcpy]
Component: XPCOM → General
Keywords: meta
QA Contact: xpcom → general
Summary: Crash [@ fastcopy_I ] → [meta] crash @ fastcopy_I | _VEC_memcpy
Depends on: 720652
(In reply to Scoobidiver from comment #2)
> Stack traces are different from the ones in comment 0 and look like:
> Frame 	Module 	Signature [Expand] 	Source
> 0 	mozcrt19.dll 	fastcopy_I 	
> 1 	mozcrt19.dll 	_VEC_memcpy 	
> 2 	mozcrt19.dll 	_VEC_memcpy 	
> 3 	xul.dll 	nsVoidArray::operator= 
> obj-firefox/xpcom/build/nsVoidArray.cpp:339

This is reported now as this signature:
fastcopy_I | _VEC_memcpy | _VEC_memcpy | nsVoidArray::operator=(nsVoidArray const&)
https://crash-stats.mozilla.com/report/list?signature=fastcopy_I%20%7C%20_VEC_memcpy%20%7C%20_VEC_memcpy%20%7C%20nsVoidArray%3A%3Aoperator%3D%28nsVoidArray%20const%26%29

Is this the same bug, i.e. should we put this into the crash signature field now?

Still, this seems to happen only up to Firefox 8, so this might actually just be fixed. If so, we probably should mark it that way.
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #32)
> Still, this seems to happen only up to Firefox 8, so this might actually
> just be fixed. If so, we probably should mark it that way.
Let's restrict this bug to this signature and close it as WFM.
Status: NEW → RESOLVED
Crash Signature: [@ fastcopy_I] [@ fastcopy_I | _VEC_memcpy] → [@ fastcopy_I] [@ fastcopy_I | _VEC_memcpy | _VEC_memcpy | nsVoidArray::operator=(nsVoidArray const&)]
Closed: 8 years ago
Keywords: meta
Resolution: --- → WORKSFORME
Summary: [meta] crash @ fastcopy_I | _VEC_memcpy → Crash @ fastcopy_I | _VEC_memcpy | _VEC_memcpy | nsVoidArray::operator=
You need to log in before you can comment on or make changes to this bug.