Closed Bug 598774 Opened 15 years ago Closed 13 years ago

Crash @ fastcopy_I | _VEC_memcpy | _VEC_memcpy | nsVoidArray::operator=

Categories

(Core :: General, defect)

Product:
Core
Component:
General
Version:
7 Branch
Platform:
x86
Windows 7
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

()

Status:
RESOLVED WORKSFORME
Tracking Flags:
Tracking Status
firefox7 + affected

People

(Reporter: scoobidiver, Unassigned)

References

Details

(Keywords: crash, regression)

Crash Data

Build : Mozilla/5.0 (Windows NT 6.1; rv:2.0b6) Gecko/20100101 Firefox/4.0b6 This is a crash signature that is present in 3.6.9, 4.0b5 and 4.0b6. It is #38 top crasher for the last 2 weeks in 4.0b6. Here are the comments about this crash signature in 4.0b5 and b6 : "Every auto launch of Firefox beta crashes. I have to manually restart it in order to get it to work" "Anytime i try to sign into google documents, it will crash" "When i started this happened always with mozilla firefox v4" "What I have noticed is the CPU usage always 100%, as soon as I start Firefox Beta 5 ... this is not good" Signature fastcopy_I UUID a3094374-bb6c-48b4-9aa2-258652100922 Time 2010-09-22 10:28:31.556488 Uptime 2352 Install Age 419386 seconds (4.9 days) since version was first installed. Product Firefox Version 4.0b6 Build ID 20100914083612 Branch 2.0 OS Windows NT OS Version 6.1.7600 CPU x86 CPU Info GenuineIntel family 6 model 23 stepping 10 Crash Reason EXCEPTION_ACCESS_VIOLATION_READ Crash Address 0xc846000 App Notes AdapterVendorID: 10de, AdapterDeviceID: 06e4 Crashing Thread Frame Module Signature [Expand] Source 0 mozcrt19.dll fastcopy_I 1 mozcrt19.dll _VEC_memcpy 2 xul.dll nsXULTooltipListener::`scalar deleting destructor' 3 mozcrt19.dll _VEC_memcpy 4 xul.dll nsTArray_base::ShiftData obj-firefox/xpcom/build/nsTArray.cpp:175 5 xul.dll nsEventListenerManager::Disconnect content/events/src/nsEventListenerManager.cpp:1234 6 xul.dll nsEventListenerManager::cycleCollection::Unlink content/events/src/nsEventListenerManager.cpp:348 7 xul.dll nsCycleCollector::CollectWhite xpcom/base/nsCycleCollector.cpp:1784 8 xul.dll nsCycleCollector::FinishCollection xpcom/base/nsCycleCollector.cpp:2655 9 xul.dll nsCycleCollector::Collect xpcom/base/nsCycleCollector.cpp:2471
Crash Signature: [@ fastcopy_I ]
This signature still garners a fair amount of crashes per week: https://crash-stats.mozilla.com/report/list?signature=fastcopy_I The user in Bug 656612 seems to hit the crash pretty frequently.
Depends on: 656612
It's #55 top crasher in 5.0, #94 in 6.0 and #11 in 7.0a2. Stack traces are different from the ones in comment 0 and look like: Frame Module Signature [Expand] Source 0 mozcrt19.dll fastcopy_I 1 mozcrt19.dll _VEC_memcpy 2 mozcrt19.dll _VEC_memcpy 3 xul.dll nsVoidArray::operator= obj-firefox/xpcom/build/nsVoidArray.cpp:339 4 xul.dll nsCOMArray_base::Clear obj-firefox/xpcom/build/nsCOMArray.cpp:175 5 xul.dll nsRefPtr<nsPresContext>::~nsRefPtr<nsPresContext> obj-firefox/dist/include/nsAutoPtr.h:969 6 xul.dll nsDocument::~nsDocument content/base/src/nsDocument.cpp:1681 7 xul.dll nsHTMLDocument::`vector deleting destructor' 8 xul.dll nsNodeUtils::LastRelease content/base/src/nsNodeUtils.cpp:326 9 xul.dll nsXMLDocument::Release content/xml/document/src/nsXMLDocument.cpp:249 10 xul.dll DoDeferredRelease<nsISupports*> js/src/xpconnect/src/xpcjsruntime.cpp:607 11 xul.dll XPCJSRuntime::GCCallback js/src/xpconnect/src/xpcjsruntime.cpp:917 ... 5.0 crash stats show a correlation with the RealPlayer Browser Record Plugin extension: 65% (75/116) vs. 2% (893/47599) {ABDE892B-13A8-4d1b-88E6-365A6E755758} 65% (75/116) vs. 1% (257/47599) 1.0.0 78% (90/116) vs. 1% (285/47599) nprndlffbrowserrecordext.dll 69% (80/116) vs. 1% (248/47599) 1.0.0.333 9% (10/116) vs. 0% (37/47599) 1.0.0.334
Summary: Firefox 4.0b6 crash [@ fastcopy_I ] → Crash [@ fastcopy_I ] mainly with the RealPlayer Browser Record Plugin extension
(In reply to Scoobidiver from comment #2) > 5.0 crash stats show a correlation with the RealPlayer Browser Record Plugin > extension: > 65% (75/116) vs. 2% (893/47599) {ABDE892B-13A8-4d1b-88E6-365A6E755758} Can we do something about that? Should we contact them? Block it? Something else?
It's #6 top browser crasher in 7.0b1.
tracking-firefox7: --- → ?
It is #93 top crasher in 6.0 and #11 in 7.0b1. There's no longer a correlation with RealPlayer in 6.0 or 7.0.
Summary: Crash [@ fastcopy_I ] mainly with the RealPlayer Browser Record Plugin extension → Crash [@ fastcopy_I ]
not something that the release drivers are going to track.
tracking-firefox7: ? → -
we were averaging about 750 crashes per day for this durning july. now averaging around 920 over the last 30 days. distribution is across many releases but it does look like a volume regression some where along the path in the recent releases. firefox 7 has almost as many crashes per day as firefox 6 with substantially fewer users. fastcopy_I date total breakdown by build crashes count build, count build, ... 20110823 754 161 6.02011081116, 153 7.02011081615, 114 3.6.202011080313, 54 5.02011061515, 40 5.0.12011070718, 27 3.6.182011061423, 23 4.0.12011041322, 19 3.6.32010040108, 14 4.02011031805, 12 5.02011042714, 10 3.6.132010120307, 9 8.0a22011082204, 8 3.62010011514, 7 4.0b102011012116, 6 4.0b72010110414, 6 3.6.152011030302, 5 5.02011052709, 5 3.6.172011042014, 5 3.6.162011031913, 4 4.0b32010080519, 4 3.6.42010051314, 3 9.0a12011082211, 3 6.02011080403, 3 6.02011071317, 3 4.0b82010121417, 3 3.6.82010072215, 3 3.6.122010102621, 3 3.5.22009072922, 3 3.5.192011042014, 3 3.0.192010031422, 2 8.0a22011082104, 2 7.0a22011081604, 2 6.02011072115, 2 5.02011061417, 2 4.0b22010072019, 2 4.0b122011022221, 2 4.0b112011020314, 2 3.6.92010082415, 2 3.6.62010062523, 2 3.6.42010061114, 2 3.6.112010101211, 2 3.6.102010091412, 1 9.0a12011082303, 1 9.0a12011082203, 1 9.0a12011082103, 1 8.0a22011082004, 1 7.0a22011081004, 1 5.02011051719, 1 4.0b62010091408, 1 4.0b12010063014, 1 3.6b52009120414, 1 3.6.72010070102, 1 3.6.3plugin12010040510, 1 3.6.22010031607, 1 3.6.192011070719, 1 3.6.142011020713, 1 3.52009061517, 1 3.0.162009120208, 1 3.0.12008070208,
tracking-firefox7: - → ?
Something definitely regressed here. I looked at some data on 7.0 beta compared to 6.0 beta. We are getting about 105 to 200 of these on release versions ie: 5.0, 6.0. for 7.0 beta we have over 200 a day. During 6.0 beta we had about 50 a day.
It's #3 top browser crasher (w/o hangs) in 7.0b2.
tracking-firefox7: ? → +
Ok, so how do we reason about this? It's coming down to the wire for 7
Keywords: regressionwindow-wanted
The crash ratio between 7.0 (about 3000 crashes/week/1M ADU) and 6.0.2 (23 crashes/week/1M ADU) is about 130. According to crash stats in Firefox 7 Aurora: https://crash-stats.mozilla.com/report/list?signature=fastcopy_I&&range_value=4&range_unit=weeks&version=Firefox:7.0a2 the regression range for the spike is: http://hg.mozilla.org/releases/mozilla-aurora/pushloghtml?fromchange=bcd26d45856a&tochange=d80597c09388 bug 678588 changed singleDeviceInterval from 24 hours to 1 hour. It has only highlighted an existing problem.
Blocks: 678588
Keywords: regressionwindow-wanted
Version: Trunk → 7 Branch
I backed out bug 678588 on mozilla-beta8 but kept it everywhere else for now.
status-firefox7: --- → unaffected
(In reply to Christian Legnitto [:LegNeato] from comment #12) > I backed out bug 678588 on mozilla-beta8 but kept it everywhere else for now. s/mozilla-beta8/mozilla-beta7/
Indeed, sorry about that...was looking at something 8 related :-/ scoobi, you are a rockstar
I've been looking through crash reports and the stack traces for fastcopy are all over the place (because fastcopy is part of the implementation for malloc, realloc, etc on Win32). I have a nagging suspicion we are dealing with a heap corruption, but it isn't obvious where it is occurring (as with most heap corruptions). I'm thinking analysis of the stack straces for common frames would be helpful. The folks in #metrics and #breakpad say that chofmann has a tool to do this. chofmann: can you explain what your stack analysis tool does and help us use it, if appropriate?
We have some other hints pointing at memory corruption introduced sometime this summer; jduell was tracking one last night responsible for ~13% of crashes on fennec that looks like mem corruption, and I've heard a few other anecdotes about hard-to-pin-down crashes pointing there.
I'm currently working on parsing *all* of the crash reports from the last 30 days (from https://crash-analysis.mozilla.com/crash_analysis/) to see if I can find any common patterns, particularly with regards to the stack frames and version numbers. I'll update the bug when I have data.
Here is the breakdown of counts of crashes in fastcopy_I grouped by version for the past 30 days: 3.0 12 3.0.1 9 3.0.10 11 3.0.11 4 3.0.13 1 3.0.14 1 3.0.15 3 3.0.16 2 3.0.17 1 3.0.19 147 3.0.3 14 3.0.4 4 3.0.5 4 3.0.6 5 3.0.7 5 3.0.8 7 3.1b3 12 3.5 44 3.5.1 1 3.5.10 3 3.5.11 5 3.5.13 2 3.5.14 1 3.5.15 4 3.5.16 9 3.5.17 2 3.5.18 6 3.5.19 36 3.5.2 71 3.5.3 12 3.5.4 4 3.5.5 26 3.5.6 11 3.5.7 6 3.5.8 3 3.5.9 9 3.5b4 2 3.6 312 3.6.10 152 3.6.11 61 3.6.12 152 3.6.13 476 3.6.14 38 3.6.15 116 3.6.16 110 3.6.17 217 3.6.18 562 3.6.19 34 3.6.2 78 3.6.20 2990 3.6.21 2179 3.6.22 3235 3.6.3 488 3.6.3plugin1 6 3.6.4 220 3.6.6 140 3.6.7 68 3.6.8 178 3.6.9 45 3.6b1 1 3.6b2 1 3.6b3 2 3.6b4 2 3.6b5 16 3.7a3 3 3.7a5 2 3.7a6pre 1 4.0 257 4.0.1 685 4.0b1 73 4.0b10 52 4.0b11 55 4.0b12 56 4.0b12pre 1 4.0b13pre 2 4.0b2 117 4.0b3 119 4.0b4 34 4.0b5 62 4.0b6 39 4.0b7 115 4.0b7pre 2 4.0b8 78 4.0b9 44 4.2a1pre 1 5.0 1722 5.0.1 540 5.0a2 1 6.0 3181 6.0.1 1705 6.0.2 3519 6.0a2 1 7.0 6389 7.0a2 56 8.0a1 15 8.0a2 438 9.0a1 108 Given the number of crashes in the released versions (up to 6.x at the time this was written), it appears we are dealing with an outstanding bug. But, the counts in the unreleased versions are troubling given the percentage of users on those versions. 7, 8, and 9 have about the same number of crashes (7006) as 6.x (8405). Anyway, I don't think there is much new data in this comment that hasn't already been said. Next up, I will be looking at the stack traces for all the crashes (31851 total) looking for frame correlations.
First, I've created some tools to enable easier crash analysis. Code available at https://github.com/mozilla-services/socorro-crash-utils. I've tried to include as much as possible in a Python module, which others should just be able to pick up and import to grasp the power. Anyway, I've fetched ~20k crash dumps so far. The 100 most common frames *anywhere in the crashed stack* are: 28114 _VEC_memcpy 24249 19960 fastcopy_I 5716 js_Interpret 5502 nsJSContext::EvaluateString(nsAString_internal const &,void *,nsIPrincipal *,char const *,unsigned int,unsigned int,nsAString_internal *,int *) 5491 nsScriptLoader::ProcessRequest(nsScriptLoadRequest *) 5477 nsScriptLoader::EvaluateScript(nsScriptLoadRequest *,nsString const &) 5405 JS_EvaluateUCScriptForPrincipals 5370 js_Execute 5350 PushBackTrackState 5321 str_match 5311 nsScriptLoader::ProcessPendingRequests() 5299 js_ExecuteRegExp 5220 DoMatch 5200 nsCOMArray_base::RemoveObject(nsISupports *) 3911 nsThread::ProcessNextEvent(int,int *) 3641 MessageLoop::RunHandler() 3450 wmain 3430 nsCOMPtr_base::~nsCOMPtr_base() 3390 nsTimerImpl::Fire() 3389 MessageLoop::Run() 3382 XRE_main 3378 mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate *) 3270 JS_GC 3261 nsVoidArray::operator=(nsVoidArray const &) 3221 nsBaseAppShell::Run() 3217 nsCOMArray_base::Clear() 3169 nsTimerEvent::Run() 3040 nsAppStartup::Run() 3014 _initterm 3014 _RTC_Initialize 2988 js_GC(JSContext *,JSCompartment *,JSGCInvocationKind) 2983 XPCJSRuntime::GCCallback(JSContext *,JSGCStatus) 2982 DoDeferredRelease<nsISupports *> 2981 DOMGCCallback 2962 JS_CompartmentGC 2943 nsXPConnect::Collect() 2917 nsNodeUtils::LastRelease(nsINode *) 2897 nsDocument::~nsDocument() 2770 BaseProcessStart 2696 nsXPConnect::GarbageCollect() 2692 nsJSContext::GarbageCollectNow() 2194 Str 2018 _MD_CURRENT_THREAD 1886 nsHTMLDocument::Release() 1760 nsHTMLDocument::`scalar deleting destructor'(unsigned int) 1547 arena_ralloc 1476 F_1799079006_______________________________________________ 1414 GetCodePageFileInfo 1371 realloc 1304 _RtlUserThreadStart 1294 JO 1066 mozilla::storage::AsyncExecuteStatements::AsyncExecuteStatements(nsTArray<mozilla::storage::StatementData,nsTArrayDefaultAllocator> &,mozilla::storage::Connection *,mozIStorageStatementCallback *) 1042 nsHTMLDocument::`vector deleting destructor'(unsigned int) 935 F_2118623609_________________________________________ 884 nsXMLDocument::Release() 877 F_1229453757___________________________________________________________ 876 PixelCopy32 734 JS_CallFunctionValue 714 arena_ralloc_large_grow 694 F_876307109____________________________ 692 F2052336212_____________________________________________________________ 682 mozilla::`anonymous namespace'::ContainerState::ProcessDisplayItems(nsDisplayList const &,mozilla::FrameLayerBuilder::Clip &) 678 F_958494385______________________ 676 JA 643 F_655827159____________________________________ 598 nsGenericElement::DestroyContent() 583 FindAtomW 554 arena_dalloc 551 js::ExternalInvoke(JSContext *,js::Value const &,js::Value const &,unsigned int,js::Value *,js::Value *) 520 F790412612___________________________ 502 nsDOMEvent::Release() 499 InterlockedExchange 496 arena_dalloc_small 495 LdrpGetShimEngineInterface 488 nsXPCWrappedJSClass::CallMethod(nsXPCWrappedJS *,unsigned short,XPTMethodDescriptor const *,nsXPTCMiniVariant *) 488 nsXPCWrappedJS::CallMethod(unsigned short,XPTMethodDescriptor const *,nsXPTCMiniVariant *) 487 SharedStub 483 js::RunScript(JSContext *,JSScript *,js::StackFrame *) 473 PrepareAndDispatch 459 js_Invoke 437 __tmainCRTStartup 428 ScopedXPCOMStartup::~ScopedXPCOMStartup() 426 LdrpAppendToForwarderList 425 F_973733276______________________________ 421 nsEventListenerManager::Disconnect() 418 nsCycleCollector::CollectWhite() 408 js::mjit::JaegerShot(JSContext *) 406 nsCycleCollector::FinishCollection() 402 F_1860184299______________________________ 397 F_1572877831_________________________________ 396 js::Invoke(JSContext *,js::CallArgs const &,js::ConstructOption) 395 F_1946984377_____________________ 394 AssignExpr 337 js::mjit::EnterMethodJIT(JSContext *,js::StackFrame *,void *,js::Value *) 329 nsGlobalWindow::RunTimeout(nsTimeout *) 320 nsDOMEvent::`scalar deleting destructor'(unsigned int) 320 malloc 317 js::Interpret(JSContext *,js::StackFrame *,unsigned int,js::InterpMode) 316 nsIFrame::InvalidateInternal(nsRect const &,int,int,nsIFrame *,unsigned int)
OK, I finished downloading all the full crash files. Of the 31851 crashes with "fastcopy_I" in their signature in the past 30 days, here are some statistics. Top 100 symbols occurring in any frame: 44640 _VEC_memcpy 37263 31850 fastcopy_I 8653 js_Interpret 8104 nsJSContext::EvaluateString(nsAString_internal const &,void *,nsIPrincipal *,char const *,unsigned int,unsigned int,nsAString_internal *,int *) 8091 nsScriptLoader::ProcessRequest(nsScriptLoadRequest *) 8064 nsScriptLoader::EvaluateScript(nsScriptLoadRequest *,nsString const &) 8012 PushBackTrackState 7958 JS_EvaluateUCScriptForPrincipals 7917 js_ExecuteRegExp 7902 js_Execute 7837 nsScriptLoader::ProcessPendingRequests() 7829 DoMatch 7828 str_match 7659 nsCOMArray_base::RemoveObject(nsISupports *) 6490 nsThread::ProcessNextEvent(int,int *) 6033 MessageLoop::RunHandler() 5632 nsTimerImpl::Fire() 5631 MessageLoop::Run() 5607 mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate *) 5449 nsVoidArray::operator=(nsVoidArray const &) 5386 nsCOMArray_base::Clear() 5383 wmain 5355 JS_GC 5333 nsBaseAppShell::Run() 5294 XRE_main 5256 nsTimerEvent::Run() 4976 js_GC(JSContext *,JSCompartment *,JSGCInvocationKind) 4971 XPCJSRuntime::GCCallback(JSContext *,JSGCStatus) 4969 DoDeferredRelease<nsISupports *> 4968 DOMGCCallback 4939 JS_CompartmentGC 4905 nsXPConnect::Collect() 4898 nsNodeUtils::LastRelease(nsINode *) 4846 nsDocument::~nsDocument() 4686 _initterm 4686 _RTC_Initialize 4671 nsAppStartup::Run() 4636 BaseProcessStart 4478 nsXPConnect::GarbageCollect() 4473 nsJSContext::GarbageCollectNow() 4420 nsCOMPtr_base::~nsCOMPtr_base() 3649 Str 3624 nsHTMLDocument::Release() 3402 _MD_CURRENT_THREAD 2531 nsHTMLDocument::`scalar deleting destructor'(unsigned int) 2441 arena_ralloc 2421 F_1799079006_______________________________________________ 2339 GetCodePageFileInfo 2210 realloc 2160 nsHTMLDocument::`vector deleting destructor'(unsigned int) 2154 JO 1757 _RtlUserThreadStart 1649 F_2118623609_________________________________________ 1473 nsRefPtr<nsPresContext>::~nsRefPtr<nsPresContext>() 1400 F_1229453757___________________________________________________________ 1398 PixelCopy32 1337 mozilla::storage::AsyncExecuteStatements::AsyncExecuteStatements(nsTArray<mozilla::storage::StatementData,nsTArrayDefaultAllocator> &,mozilla::storage::Connection *,mozIStorageStatementCallback *) 1155 JS_CallFunctionValue 1151 F_876307109____________________________ 1148 F2052336212_____________________________________________________________ 1133 F_958494385______________________ 1124 arena_ralloc_large_grow 1122 nsAppShell::Run() 1113 JA 1058 F_655827159____________________________________ 1039 mozilla::`anonymous namespace'::ContainerState::ProcessDisplayItems(nsDisplayList const &,mozilla::FrameLayerBuilder::Clip &) 1027 nsXMLDocument::Release() 1024 FindAtomW 929 nsGenericElement::DestroyContent() 914 arena_dalloc 890 arena_dalloc_small 876 js::ExternalInvoke(JSContext *,js::Value const &,js::Value const &,unsigned int,js::Value *,js::Value *) 836 InterlockedExchange 825 nsXPCWrappedJS::CallMethod(unsigned short,XPTMethodDescriptor const *,nsXPTCMiniVariant *) 822 nsXPCWrappedJSClass::CallMethod(nsXPCWrappedJS *,unsigned short,XPTMethodDescriptor const *,nsXPTCMiniVariant *) 822 js_Invoke 822 SharedStub 803 PrepareAndDispatch 782 F790412612___________________________ 767 js::RunScript(JSContext *,JSScript *,js::StackFrame *) 763 nsDOMEvent::Release() 713 ScopedXPCOMStartup::~ScopedXPCOMStartup() 704 F_973733276______________________________ 698 __tmainCRTStartup 677 F_1860184299______________________________ 665 LdrpGetShimEngineInterface 662 nsEventListenerManager::Disconnect() 652 F_1946984377_____________________ 650 js::mjit::JaegerShot(JSContext *) 641 js::Invoke(JSContext *,js::CallArgs const &,js::ConstructOption) 614 F_1572877831_________________________________ 610 nsCycleCollector::CollectWhite() 593 nsCycleCollector::FinishCollection() 575 AssignExpr 556 LdrpAppendToForwarderList 554 js::mjit::EnterMethodJIT(JSContext *,js::StackFrame *,void *,js::Value *) 554 js::Interpret(JSContext *,js::StackFrame *,unsigned int,js::InterpMode) 551 nsDOMEvent::`scalar deleting destructor'(unsigned int) 509 malloc Top 50 symbols in any frame for stacks containing "malloc": 152 PresShell::Paint(nsIView *,nsIWidget *,nsRegion const &,nsIntRegion const &,int,int) 152 nsDisplayList::PaintForFrame(nsDisplayListBuilder *,nsRenderingContext *,nsIFrame *,unsigned int) 155 DispatchClientMessage 155 KiUserCallbackDispatcher 158 nsJSContext::EvaluateString(nsAString_internal const &,void *,nsIPrincipal *,char const *,unsigned int,unsigned int,nsAString_internal *,int *) 159 nsWindow::DispatchWindowEvent(nsGUIEvent *,nsEventStatus &) 160 PresShell::FlushPendingNotifications(mozFlushType) 160 nsWindow::OnPaint(HDC__ *,unsigned int) 164 nsWindow::DispatchEvent(nsGUIEvent *,nsEventStatus &) 166 nsViewManager::DispatchEvent(nsGUIEvent *,nsIView *,nsEventStatus *) 170 arena_ralloc_large_grow 173 PrepareAndDispatch 173 UserCallWinProcCheckWow 174 nsWindow::ProcessMessage(unsigned int,unsigned int &,long &,long *) 176 JS_CallFunctionValue 176 __tmainCRTStartup 177 SharedStub 177 nsXPCWrappedJS::CallMethod(unsigned short,XPTMethodDescriptor const *,nsXPTCMiniVariant *) 180 nsWindow::WindowProc(HWND__ *,unsigned int,unsigned int,long) 180 nsXPCWrappedJSClass::CallMethod(nsXPCWrappedJS *,unsigned short,XPTMethodDescriptor const *,nsXPTCMiniVariant *) 183 nsWindow::WindowProcInternal(HWND__ *,unsigned int,unsigned int,long) 184 InternalCallWinProc 185 __fnDWORD 190 _MD_CURRENT_THREAD 190 nsAppStartup::Run() 192 GetCodePageFileInfo 200 XRE_main 201 mozilla::FrameLayerBuilder::BuildContainerLayerFor(nsDisplayListBuilder *,mozilla::layers::LayerManager *,nsIFrame *,nsDisplayItem *,nsDisplayList const &) 205 wmain 214 CallWindowProcCrashProtected 223 nsBaseAppShell::Run() 234 MessageLoop::Run() 260 JA 266 nsTimerEvent::Run() 274 nsTimerImpl::Fire() 314 js_Invoke 329 arena_ralloc 364 realloc 402 js_Interpret 404 BaseProcessStart 457 mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate *) 482 JO 489 MessageLoop::RunHandler() 493 nsThread::ProcessNextEvent(int,int *) 509 malloc 722 mozilla::`anonymous namespace'::ContainerState::ProcessDisplayItems(nsDisplayList const &,mozilla::FrameLayerBuilder::Clip &) 826 Str 1161 fastcopy_I 1456 _VEC_memcpy 2863 Note how different this list is from the global one! Top 100 {symbol, stack position} pairs (can kinda group related stacks by the frame's location in the stack): 20601 0 fastcopy_I 20426 1 _VEC_memcpy 11370 2 _VEC_memcpy 9074 0 fastcopy_I 8772 1 _VEC_memcpy 6607 3 PushBackTrackState 6533 4 js_ExecuteRegExp 6487 5 DoMatch 6439 6 str_match 6316 7 js_Interpret 6234 12 nsScriptLoader::ProcessRequest(nsScriptLoadRequest *) 6234 10 nsJSContext::EvaluateString(nsAString_internal const &,void *,nsIPrincipal *,char const *,unsigned int,unsigned int,nsAString_internal *,int *) 6233 11 nsScriptLoader::EvaluateScript(nsScriptLoadRequest *,nsString const &) 6217 9 JS_EvaluateUCScriptForPrincipals 6217 8 js_Execute 6157 13 nsCOMArray_base::RemoveObject(nsISupports *) 6126 14 nsScriptLoader::ProcessPendingRequests() 5037 3 nsVoidArray::operator=(nsVoidArray const &) 4959 4 nsCOMArray_base::Clear() 4358 6 nsDocument::~nsDocument() 3910 8 nsNodeUtils::LastRelease(nsINode *) 3257 11 XPCJSRuntime::GCCallback(JSContext *,JSGCStatus) 3256 10 DoDeferredRelease<nsISupports *> 3254 12 DOMGCCallback 3223 13 js_GC(JSContext *,JSCompartment *,JSGCInvocationKind) 3221 15 JS_GC 3219 14 JS_CompartmentGC 3205 5 nsCOMPtr_base::~nsCOMPtr_base() 3055 16 nsXPConnect::Collect() 2950 9 nsHTMLDocument::Release() 2939 21 nsThread::ProcessNextEvent(int,int *) 2927 19 nsTimerImpl::Fire() 2922 20 nsTimerEvent::Run() 2883 18 nsJSContext::GarbageCollectNow() 2883 17 nsXPConnect::GarbageCollect() 2871 24 MessageLoop::RunHandler() 2404 2 F_1799079006_______________________________________________ 2104 7 nsHTMLDocument::`scalar deleting destructor'(unsigned int) 2049 0 fastcopy_I 2043 1 _VEC_memcpy 1942 31 1869 2 1858 23 mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate *) 1802 7 nsHTMLDocument::`vector deleting destructor'(unsigned int) 1734 25 MessageLoop::Run() 1723 27 1706 26 nsBaseAppShell::Run() 1691 29 XRE_main 1690 30 wmain 1688 28 nsAppStartup::Run() 1687 22 _MD_CURRENT_THREAD 1655 33 _initterm 1655 32 _RTC_Initialize 1573 37 1570 2 F_2118623609_________________________________________ 1560 40 1500 36 BaseProcessStart 1489 26 MessageLoop::Run() 1339 23 1314 34 1313 25 1311 3 _VEC_memcpy 1252 28 nsBaseAppShell::Run() 1224 4 realloc 1220 2 arena_ralloc 1211 3 arena_ralloc 1201 35 1184 22 mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate *) 1121 2 arena_ralloc_large_grow 1102 3 F_876307109____________________________ 1100 4 F_958494385______________________ 1095 5 F2052336212_____________________________________________________________ 1093 36 1082 5 nsRefPtr<nsPresContext>::~nsRefPtr<nsPresContext>() 1054 39 984 6 F_655827159____________________________________ 970 33 949 3 realloc 902 2 880 30 nsAppStartup::Run() 870 32 wmain 860 9 nsXMLDocument::Release() 855 35 _initterm 855 34 _RTC_Initialize 827 31 XRE_main 804 27 mozilla::storage::AsyncExecuteStatements::AsyncExecuteStatements(nsTArray<mozilla::storage::StatementData,nsTArrayDefaultAllocator> &,mozilla::storage::Connection *,mozIStorageStatementCallback *) 796 5 784 3 js_ExecuteRegExp 777 4 F_1229453757___________________________________________________________ 777 3 PixelCopy32 776 13 nsScriptLoader::ProcessPendingRequests() 774 2 PushBackTrackState 774 2 F790412612___________________________ 767 4 DoMatch 764 5 str_match 758 6 js_Interpret 747 11 nsScriptLoader::ProcessRequest(nsScriptLoadRequest *) 744 8 JS_EvaluateUCScriptForPrincipals 744 7 js_Execute 742 9 nsJSContext::EvaluateString(nsAString_internal const &,void *,nsIPrincipal *,char const *,unsigned int,unsigned int,nsAString_internal *,int *) When looking at these, keep in mind we're likely looking at multiple bugs.
I have ~80k crash dumps with fastcopy_I from the past 90 days harvested on my computer. But, I really don't know where to continue analysis beyond the primitive ones pasted above. If anyone wants me to run a custom report, I can do that (it does take a while to fetch all the data from crash-stats). I just need to know what to run. In hopes of reproducing the crashes, I wrote an extension that surfs the web for you, hopefully generating entropy to trigger the crash. It is kinda hacky, but works well enough: https://github.com/mozilla-services/sync-spray. Unfortunately, after running Nightly for 30 minutes on a Windows box, I was unable to get a crash. I'll keep on it and will hopefully get lucky.
Here's a breakdown of the crash counts by version and build date for the last 90 days or so: 3.0 2008-05-29 06:00:00 24 3.0 Total 24 3.0.1 2008-07-02 08:00:00 28 3.0.1 Total 28 3.0.10 2009-04-23 16:00:00 17 3.0.10 Total 17 3.0.11 2009-06-02 15:00:00 11 3.0.11 Total 11 3.0.12 2009-07-06 11:00:00 1 3.0.12 Total 1 3.0.13 2009-07-30 22:00:00 2 3.0.13 Total 2 3.0.14 2009-08-27 07:00:00 1 3.0.14 Total 1 3.0.15 2009-10-16 01:00:00 10 3.0.15 Total 10 3.0.16 2009-12-02 08:00:00 2 3.0.16 Total 2 3.0.17 2009-12-21 16:00:00 3 3.0.17 Total 3 3.0.18 2010-02-02 20:00:00 4 3.0.18 Total 4 3.0.19 2010-03-14 22:00:00 413 3.0.19 Total 413 3.0.3 2008-09-24 17:00:00 20 3.0.3 Total 20 3.0.4 2008-10-29 20:00:00 7 3.0.4 Total 7 3.0.5 2008-12-01 22:00:00 30 3.0.5 Total 30 3.0.6 2009-01-19 13:00:00 11 3.0.6 Total 11 3.0.7 2009-02-19 10:00:00 20 3.0.7 Total 20 3.0.8 2009-03-26 09:00:00 15 3.0.8 Total 15 3.0b4 2008-03-07 14:00:00 1 3.0b4 Total 1 3.0b5 2008-03-26 20:00:00 1 3.0b5 Total 1 3.1b1 2008-10-07 14:00:00 1 3.1b1 Total 1 3.1b3 2009-03-04 23:00:00 19 3.1b3 2009-03-05 15:00:00 1 3.1b3 Total 20 3.5 2009-06-15 17:00:00 2 3.5 2009-06-16 22:00:00 2 3.5 2009-06-24 02:00:00 97 3.5 Total 101 3.5.1 2009-07-15 09:00:00 7 3.5.1 Total 7 3.5.10 2010-05-04 09:00:00 7 3.5.10 Total 7 3.5.11 2010-07-01 02:00:00 17 3.5.11 Total 17 3.5.13 2010-09-14 13:00:00 6 3.5.13 Total 6 3.5.14 2010-10-01 17:00:00 2 3.5.14 Total 2 3.5.15 2010-10-26 20:00:00 9 3.5.15 Total 9 3.5.16 2010-11-30 07:00:00 26 3.5.16 Total 26 3.5.17 2011-01-21 15:00:00 12 3.5.17 Total 12 3.5.18 2011-03-19 14:00:00 17 3.5.18 Total 17 3.5.19 2011-04-20 14:00:00 110 3.5.19 Total 110 3.5.2 2009-07-29 22:00:00 114 3.5.2 Total 114 3.5.3 2009-08-24 10:00:00 38 3.5.3 Total 38 3.5.4 2009-10-16 09:00:00 15 3.5.4 Total 15 3.5.5 2009-11-02 15:00:00 57 3.5.5 Total 57 3.5.5pre 2009-10-17 04:00:00 3 3.5.5pre Total 3 3.5.6 2009-12-01 22:00:00 28 3.5.6 Total 28 3.5.7 2009-12-21 16:00:00 25 3.5.7 Total 25 3.5.8 2010-02-02 16:00:00 11 3.5.8 Total 11 3.5.8pre 2010-01-13 04:00:00 1 3.5.8pre Total 1 3.5.9 2010-03-15 08:00:00 21 3.5.9 Total 21 3.5b4 2009-04-23 20:00:00 6 3.5b4 Total 6 3.5b99 2009-06-05 01:00:00 1 3.5b99 Total 1 3.6 2010-01-05 21:00:00 4 3.6 2010-01-15 14:00:00 828 3.6 Total 832 3.6.10 2010-09-14 12:00:00 402 3.6.10 Total 402 3.6.11 2010-09-30 13:00:00 5 3.6.11 2010-10-01 08:00:00 14 3.6.11 2010-10-12 11:00:00 83 3.6.11 Total 102 3.6.12 2010-10-26 21:00:00 352 3.6.12 Total 352 3.6.13 2010-11-22 05:00:00 15 3.6.13 2010-11-30 06:00:00 1 3.6.13 2010-12-03 07:00:00 1167 3.6.13 Total 1183 3.6.14 2011-01-21 14:00:00 3 3.6.14 2011-02-07 13:00:00 2 3.6.14 2011-02-18 12:00:00 78 3.6.14 Total 83 3.6.15 2011-03-03 02:00:00 307 3.6.15 Total 307 3.6.16 2011-03-19 13:00:00 319 3.6.16 Total 319 3.6.17 2011-04-14 08:00:00 31 3.6.17 2011-04-20 14:00:00 647 3.6.17 Total 678 3.6.18 2011-06-13 02:00:00 28 3.6.18 2011-06-14 23:00:00 9148 3.6.18 Total 9176 3.6.19 2011-07-07 19:00:00 555 3.6.19 Total 555 3.6.19pre 2011-07-01 03:00:00 1 3.6.19pre Total 1 3.6.2 2010-03-16 07:00:00 262 3.6.2 Total 262 3.6.20 2011-08-03 13:00:00 3775 3.6.20 Total 3775 3.6.20pre 2011-08-02 03:00:00 2 3.6.20pre Total 2 3.6.21 2011-08-30 09:00:00 2188 3.6.21 Total 2188 3.6.22 2011-09-02 13:00:00 3442 3.6.22 Total 3442 3.6.23 2011-09-20 07:00:00 1 3.6.23 Total 1 3.6.3 2010-04-01 08:00:00 1305 3.6.3 Total 1305 3.6.3plugin1 2010-04-05 10:00:00 19 3.6.3plugin1 Total 19 3.6.4 2010-04-13 17:00:00 20 3.6.4 2010-05-02 22:00:00 1 3.6.4 2010-05-03 12:00:00 3 3.6.4 2010-05-13 14:00:00 312 3.6.4 2010-05-23 18:00:00 7 3.6.4 2010-05-27 09:00:00 56 3.6.4 2010-06-11 14:00:00 152 3.6.4 Total 551 3.6.5pre 2010-05-23 04:00:00 1 3.6.5pre Total 1 3.6.6 2010-06-25 23:00:00 332 3.6.6 Total 332 3.6.7 2010-07-01 02:00:00 74 3.6.7 2010-07-13 13:00:00 58 3.6.7 Total 132 3.6.8 2010-07-22 15:00:00 444 3.6.8 Total 444 3.6.9 2010-08-24 15:00:00 96 3.6.9 Total 96 3.6b1 2009-10-14 10:00:00 1 3.6b1 Total 1 3.6b2 2009-11-08 18:00:00 2 3.6b2 Total 2 3.6b3 2009-11-15 18:00:00 2 3.6b3 Total 2 3.6b4 2009-11-24 21:00:00 7 3.6b4 Total 7 3.6b5 2009-12-04 14:00:00 38 3.6b5 Total 38 3.7a2 2010-02-28 16:00:00 4 3.7a2 Total 4 3.7a3 2010-03-15 09:00:00 6 3.7a3 Total 6 3.7a4 2010-04-07 11:00:00 1 3.7a4 Total 1 3.7a5 2010-06-10 06:00:00 6 3.7a5 Total 6 3.7a5pre 2010-05-11 04:00:00 1 3.7a5pre 2010-06-07 03:00:00 1 3.7a5pre Total 2 3.7a6pre 2010-06-18 04:00:00 3 3.7a6pre Total 3 4.0 2011-03-03 19:00:00 45 4.0 2011-03-18 05:00:00 932 4.0 Total 977 4.0.1 2011-04-13 22:00:00 3484 4.0.1 Total 3484 4.0b1 2010-06-28 21:00:00 6 4.0b1 2010-06-30 14:00:00 237 4.0b1 Total 243 4.0b10 2011-01-21 16:00:00 158 4.0b10 Total 158 4.0b11 2011-02-02 14:00:00 1 4.0b11 2011-02-03 14:00:00 213 4.0b11 Total 214 4.0b12 2011-02-22 21:00:00 212 4.0b12 Total 212 4.0b12pre 2011-02-10 03:00:00 1 4.0b12pre 2011-02-14 03:00:00 1 4.0b12pre Total 2 4.0b13pre 2011-03-20 03:00:00 1 4.0b13pre 2011-03-22 03:00:00 5 4.0b13pre Total 6 4.0b2 2010-07-20 19:00:00 209 4.0b2 Total 209 4.0b3 2010-08-04 19:00:00 1 4.0b3 2010-08-05 19:00:00 185 4.0b3 Total 186 4.0b4 2010-08-18 13:00:00 114 4.0b4 Total 114 4.0b5 2010-08-31 08:00:00 116 4.0b5 Total 116 4.0b6 2010-09-13 23:00:00 2 4.0b6 2010-09-14 08:00:00 145 4.0b6 Total 147 4.0b7 2010-11-04 14:00:00 316 4.0b7 Total 316 4.0b7pre 2010-10-02 04:00:00 4 4.0b7pre Total 4 4.0b8 2010-12-14 17:00:00 219 4.0b8 Total 219 4.0b9 2011-01-10 19:00:00 132 4.0b9 Total 132 4.2a1pre 2011-04-03 03:00:00 1 4.2a1pre 2011-04-11 03:00:00 1 4.2a1pre Total 2 5.0 2011-04-27 14:00:00 509 5.0 2011-05-17 19:00:00 322 5.0 2011-05-27 09:00:00 292 5.0 2011-06-03 10:00:00 1 5.0 2011-06-08 15:00:00 144 5.0 2011-06-13 16:00:00 36 5.0 2011-06-14 17:00:00 240 5.0 2011-06-15 15:00:00 19380 5.0 Total 20924 5.0.1 2011-07-07 18:00:00 3463 5.0.1 Total 3463 5.0a2 2011-04-30 04:00:00 1 5.0a2 2011-05-14 04:00:00 1 5.0a2 2011-05-18 04:00:00 1 5.0a2 2011-05-23 04:00:00 1 5.0a2 Total 4 6.0 2011-07-05 19:00:00 434 6.0 2011-07-13 17:00:00 791 6.0 2011-07-21 15:00:00 560 6.0 2011-07-29 08:00:00 239 6.0 2011-08-04 03:00:00 499 6.0 2011-08-11 16:00:00 4473 6.0 Total 6996 6.0.1 2011-08-30 09:00:00 1709 6.0.1 Total 1709 6.0.2 2011-08-31 12:00:00 1 6.0.2 2011-09-02 13:00:00 3815 6.0.2 Total 3816 6.0a1 2011-04-19 03:00:00 2 6.0a1 Total 2 6.0a2 2011-05-27 04:00:00 8 6.0a2 2011-06-03 04:00:00 1 6.0a2 2011-06-10 04:00:00 1 6.0a2 2011-06-12 04:00:00 1 6.0a2 2011-06-19 04:00:00 1 6.0a2 2011-06-21 04:00:00 1 6.0a2 2011-06-23 04:00:00 2 6.0a2 2011-06-24 04:00:00 1 6.0a2 2011-06-25 04:00:00 3 6.0a2 2011-06-26 04:00:00 3 6.0a2 2011-06-27 04:00:00 4 6.0a2 2011-06-28 04:00:00 3 6.0a2 2011-06-29 04:00:00 1 6.0a2 2011-06-30 04:00:00 6 6.0a2 2011-07-01 04:00:00 6 6.0a2 2011-07-02 04:00:00 6 6.0a2 2011-07-03 04:00:00 4 6.0a2 2011-07-04 04:00:00 3 6.0a2 2011-07-05 04:00:00 11 6.0a2 Total 66 7.0 2011-08-16 15:00:00 1701 7.0 2011-08-24 17:00:00 3162 7.0 2011-08-30 10:00:00 173 7.0 2011-08-31 12:00:00 1 7.0 2011-09-02 16:00:00 305 7.0 2011-09-08 13:00:00 1521 7.0 2011-09-16 09:00:00 296 7.0 Total 7159 7.0a1 2011-06-18 03:00:00 1 7.0a1 2011-06-24 03:00:00 1 7.0a1 2011-06-25 03:00:00 4 7.0a1 2011-06-28 03:00:00 3 7.0a1 2011-06-29 03:00:00 2 7.0a1 2011-07-01 03:00:00 1 7.0a1 2011-07-02 03:00:00 1 7.0a1 2011-07-03 03:00:00 2 7.0a1 2011-07-04 03:00:00 2 7.0a1 2011-07-05 03:00:00 3 7.0a1 Total 20 7.0a2 2011-07-06 04:00:00 9 7.0a2 2011-07-07 04:00:00 10 7.0a2 2011-07-08 04:00:00 12 7.0a2 2011-07-09 04:00:00 10 7.0a2 2011-07-10 04:00:00 6 7.0a2 2011-07-11 04:00:00 13 7.0a2 2011-07-12 04:00:00 16 7.0a2 2011-07-13 04:00:00 18 7.0a2 2011-07-14 04:00:00 10 7.0a2 2011-07-15 04:00:00 15 7.0a2 2011-07-16 04:00:00 12 7.0a2 2011-07-17 04:00:00 12 7.0a2 2011-07-18 04:00:00 15 7.0a2 2011-07-19 04:00:00 16 7.0a2 2011-07-20 04:00:00 21 7.0a2 2011-07-20 22:00:00 21 7.0a2 2011-07-22 04:00:00 3 7.0a2 2011-07-22 12:00:00 8 7.0a2 2011-07-23 04:00:00 14 7.0a2 2011-07-24 04:00:00 18 7.0a2 2011-07-25 04:00:00 9 7.0a2 2011-07-26 04:00:00 17 7.0a2 2011-07-27 04:00:00 13 7.0a2 2011-07-28 04:00:00 18 7.0a2 2011-07-29 04:00:00 14 7.0a2 2011-07-30 04:00:00 11 7.0a2 2011-07-31 04:00:00 17 7.0a2 2011-08-01 04:00:00 14 7.0a2 2011-08-02 04:00:00 11 7.0a2 2011-08-03 04:00:00 18 7.0a2 2011-08-04 04:00:00 19 7.0a2 2011-08-05 04:00:00 14 7.0a2 2011-08-06 04:00:00 20 7.0a2 2011-08-07 04:00:00 16 7.0a2 2011-08-08 04:00:00 9 7.0a2 2011-08-09 04:00:00 9 7.0a2 2011-08-10 04:00:00 24 7.0a2 2011-08-12 04:00:00 20 7.0a2 2011-08-13 04:00:00 10 7.0a2 2011-08-14 04:00:00 21 7.0a2 2011-08-15 04:00:00 15 7.0a2 2011-08-16 04:00:00 102 7.0a2 Total 680 8.0a1 2011-07-06 03:00:00 6 8.0a1 2011-07-07 03:00:00 2 8.0a1 2011-07-08 03:00:00 7 8.0a1 2011-07-09 03:00:00 4 8.0a1 2011-07-10 03:00:00 1 8.0a1 2011-07-11 03:00:00 6 8.0a1 2011-07-12 03:00:00 7 8.0a1 2011-07-13 03:00:00 6 8.0a1 2011-07-14 03:00:00 4 8.0a1 2011-07-15 03:00:00 9 8.0a1 2011-07-16 03:00:00 4 8.0a1 2011-07-17 03:00:00 4 8.0a1 2011-07-18 03:00:00 7 8.0a1 2011-07-19 03:00:00 3 8.0a1 2011-07-20 03:00:00 4 8.0a1 2011-07-21 03:00:00 4 8.0a1 2011-07-22 03:00:00 6 8.0a1 2011-07-23 03:00:00 4 8.0a1 2011-07-24 03:00:00 2 8.0a1 2011-07-25 03:00:00 3 8.0a1 2011-07-26 03:00:00 4 8.0a1 2011-07-27 03:00:00 4 8.0a1 2011-07-27 08:00:00 5 8.0a1 2011-07-28 03:00:00 4 8.0a1 2011-07-29 03:00:00 2 8.0a1 2011-07-30 03:00:00 5 8.0a1 2011-07-31 03:00:00 4 8.0a1 2011-08-01 03:00:00 4 8.0a1 2011-08-02 03:00:00 6 8.0a1 2011-08-04 03:00:00 4 8.0a1 2011-08-05 03:00:00 2 8.0a1 2011-08-06 03:00:00 1 8.0a1 2011-08-07 03:00:00 4 8.0a1 2011-08-08 03:00:00 4 8.0a1 2011-08-09 03:00:00 5 8.0a1 2011-08-10 03:00:00 2 8.0a1 2011-08-11 03:00:00 3 8.0a1 2011-08-12 03:00:00 2 8.0a1 2011-08-13 03:00:00 6 8.0a1 2011-08-14 03:00:00 3 8.0a1 2011-08-15 03:00:00 5 8.0a1 2011-08-16 03:00:00 10 8.0a1 Total 182 8.0a2 2011-08-17 04:00:00 1 8.0a2 2011-08-18 04:00:00 1 8.0a2 2011-08-19 04:00:00 14 8.0a2 2011-08-20 04:00:00 18 8.0a2 2011-08-21 04:00:00 15 8.0a2 2011-08-22 04:00:00 31 8.0a2 2011-08-24 04:00:00 26 8.0a2 2011-08-25 04:00:00 22 8.0a2 2011-08-26 04:00:00 14 8.0a2 2011-08-27 04:00:00 10 8.0a2 2011-08-28 04:00:00 21 8.0a2 2011-08-29 04:00:00 13 8.0a2 2011-08-30 04:00:00 9 8.0a2 2011-08-31 04:00:00 15 8.0a2 2011-09-01 04:00:00 15 8.0a2 2011-09-02 04:00:00 13 8.0a2 2011-09-03 04:00:00 14 8.0a2 2011-09-04 04:00:00 20 8.0a2 2011-09-05 04:00:00 17 8.0a2 2011-09-06 04:00:00 9 8.0a2 2011-09-07 04:00:00 9 8.0a2 2011-09-08 04:00:00 14 8.0a2 2011-09-09 04:00:00 17 8.0a2 2011-09-10 04:00:00 12 8.0a2 2011-09-11 04:00:00 11 8.0a2 2011-09-12 04:00:00 7 8.0a2 2011-09-13 04:00:00 10 8.0a2 2011-09-14 04:00:00 21 8.0a2 2011-09-15 04:00:00 24 8.0a2 2011-09-16 04:00:00 14 8.0a2 2011-09-17 04:00:00 20 8.0a2 2011-09-18 04:00:00 15 8.0a2 2011-09-19 04:00:00 8 8.0a2 2011-09-20 04:00:00 6 8.0a2 Total 486 9.0a1 2011-08-18 03:00:00 2 9.0a1 2011-08-19 03:00:00 1 9.0a1 2011-08-20 03:00:00 4 9.0a1 2011-08-21 03:00:00 4 9.0a1 2011-08-22 03:00:00 1 9.0a1 2011-08-22 11:00:00 5 9.0a1 2011-08-23 03:00:00 4 9.0a1 2011-08-24 03:00:00 2 9.0a1 2011-08-25 03:00:00 2 9.0a1 2011-08-26 03:00:00 5 9.0a1 2011-08-27 03:00:00 5 9.0a1 2011-08-28 03:00:00 4 9.0a1 2011-08-29 05:00:00 6 9.0a1 2011-08-31 03:00:00 4 9.0a1 2011-09-01 03:00:00 5 9.0a1 2011-09-02 07:00:00 3 9.0a1 2011-09-03 03:00:00 2 9.0a1 2011-09-04 03:00:00 4 9.0a1 2011-09-05 03:00:00 1 9.0a1 2011-09-06 03:00:00 2 9.0a1 2011-09-07 03:00:00 2 9.0a1 2011-09-08 03:00:00 3 9.0a1 2011-09-09 03:00:00 1 9.0a1 2011-09-10 03:00:00 7 9.0a1 2011-09-11 03:00:00 6 9.0a1 2011-09-12 03:00:00 3 9.0a1 2011-09-13 03:00:00 2 9.0a1 2011-09-14 03:00:00 4 9.0a1 2011-09-15 03:00:00 11 9.0a1 2011-09-16 07:00:00 1 9.0a1 2011-09-17 03:00:00 4 9.0a1 2011-09-17 12:00:00 2 9.0a1 2011-09-18 03:00:00 3 9.0a1 2011-09-19 03:00:00 5 9.0a1 2011-09-20 03:00:00 1 9.0a1 2011-09-20 08:00:00 4 9.0a1 2011-09-21 03:00:00 5 9.0a1 Total 130
(In reply to Christian Legnitto [:LegNeato] from comment #12) > I backed out bug 678588 For an unknown reason, the backing out hasn't prevented the spike, so it is still #3 top browser crasher without hangs in 7.0b6.
status-firefox7: unaffected → affected
Keywords: regression
After further investigations, I found that the spike happened in two close times: * a first one at the end of 7.0a2 (see comment 11) that made it #6 top browser crasher in 7.0b1. * a second one between 7.0b1 and 7.0b2 that made it #3 top browser crasher in 7.0b2 and above. The regression range for this second spike is: http://hg.mozilla.org/releases/mozilla-beta/pushloghtml?fromchange=00f56713eab6&tochange=4ee096ccdd5a
It scares me that bug 678422 (the nsPresArena over-allocating clownshoes one) is in that window. If we are dealing with heap corruption, it is possible those clownshoes were buffering us from a minefield of corrupted bits.
Forget about comment 24. It's because #3, #4 and #5 top browser crashers in 7.0b1 have been fixed in 7.0b2 or were startup crashes. The ratio between #1 top browser crasher and this crash signature has been pretty constant along the Beta phase, i.e. about 2.
Here are crash stats from the past 14 days: Build Percentage Count Rank 8.0a2 0.65 145 11 7.0b6 1.03 868 10 7.0b5 0.85 1268 10 6.0.2 0.11 1628 109 According to the mozilla-beta commit logs (https://hg.mozilla.org/releases/mozilla-beta/shortlog/), the backout of bug 678588 was the last commit before 7.0b6 was cut. Unfortunately, the percentage of crashes went down marginally (at least compared to the 6.0.2 0.11% baseline). If the percentage holds up, it looks like fastcopy_I is going to be a top crasher on stable/7.0 starting tomorrow :(
I backed out bug 678588 in beta8 as well. If it is determined we don't need to do that we can reland
(In reply to Scoobidiver from comment #23) > (In reply to Christian Legnitto [:LegNeato] from comment #12) > > I backed out bug 678588 > For an unknown reason, the backing out hasn't prevented the spike, so it is > still #3 top browser crasher without hangs in 7.0b6. Actually, because of this investigation we are going to re-land bug 678588 on beta8.
(In reply to Scoobidiver from comment #23) > For an unknown reason, the backing out hasn't prevented the spike I think I understood. There's a delay to update before the release of a new Aurora version so every crashes in the latest Aurora 7 build were more important than in the previous ones, making think it was a spike. 7.0a1: 20 crashes/1M ADU/day 7.0a2: 130 crashes/1M ADU/day The regression range could be in the changesets between 7.0a1/20110705 and 7.0a2/20110706: http://hg.mozilla.org/releases/mozilla-aurora/pushloghtml?fromchange=cc6e30cce8af&tochange=92120afb1a58
No longer blocks: 678588
Keywords: topcrash
Depends on: 687672
This is pretty far down on the list for 10.0 #64 and 9.0.1 #94. I am going to take it off the top crash list.
Keywords: topcrash
Crash Signature: [@ fastcopy_I ] → [@ fastcopy_I] [@ fastcopy_I | _VEC_memcpy]
Component: XPCOM → General
Keywords: meta
QA Contact: xpcom → general
Summary: Crash [@ fastcopy_I ] → [meta] crash @ fastcopy_I | _VEC_memcpy
Depends on: 720652
(In reply to Scoobidiver from comment #2) > Stack traces are different from the ones in comment 0 and look like: > Frame Module Signature [Expand] Source > 0 mozcrt19.dll fastcopy_I > 1 mozcrt19.dll _VEC_memcpy > 2 mozcrt19.dll _VEC_memcpy > 3 xul.dll nsVoidArray::operator= > obj-firefox/xpcom/build/nsVoidArray.cpp:339 This is reported now as this signature: fastcopy_I | _VEC_memcpy | _VEC_memcpy | nsVoidArray::operator=(nsVoidArray const&) https://crash-stats.mozilla.com/report/list?signature=fastcopy_I%20%7C%20_VEC_memcpy%20%7C%20_VEC_memcpy%20%7C%20nsVoidArray%3A%3Aoperator%3D%28nsVoidArray%20const%26%29 Is this the same bug, i.e. should we put this into the crash signature field now? Still, this seems to happen only up to Firefox 8, so this might actually just be fixed. If so, we probably should mark it that way.
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #32) > Still, this seems to happen only up to Firefox 8, so this might actually > just be fixed. If so, we probably should mark it that way. Let's restrict this bug to this signature and close it as WFM.
Status: NEW → RESOLVED
Crash Signature: [@ fastcopy_I] [@ fastcopy_I | _VEC_memcpy] → [@ fastcopy_I] [@ fastcopy_I | _VEC_memcpy | _VEC_memcpy | nsVoidArray::operator=(nsVoidArray const&)]
Closed: 13 years ago
Keywords: meta
Resolution: --- → WORKSFORME
Summary: [meta] crash @ fastcopy_I | _VEC_memcpy → Crash @ fastcopy_I | _VEC_memcpy | _VEC_memcpy | nsVoidArray::operator=
You need to log in before you can comment on or make changes to this bug.