Closed Bug 786153 Opened 7 years ago Closed 7 years ago

crash in nsBaseAppShell::OnProcessNextEvent with Trusteer Rapport

Categories

(Firefox :: Extension Compatibility, defect, critical)

20 Branch
x86
Windows NT
defect
Not set
critical

Tracking

()

RESOLVED FIXED
Tracking Status
firefox20 + fixed

People

(Reporter: marcia, Unassigned)

References

Details

(4 keywords, Whiteboard: [fixed in Trusteer Rapport 3.5.1208.34])

Crash Data

Attachments

(1 file)

This bug was filed from the Socorro interface and is 
report bp-2ec9052c-2893-4f0d-bc13-429db2120827 .
============================================================= 

Seen while looking at crash stats. This one moved up the ranks of Beta 6 during the course of the day and now sits as #36 top crash in Beta 6: https://crash-stats.mozilla.com/report/list?signature=nsBaseAppShell::OnProcessNextEvent%28nsIThreadInternal*,%20bool,%20unsigned%20int%29

ATM I don't see any crashes in Firefox 15 final (buildID=20120824154833), but filing to keep on eye on this.

nsBaseAppShell::OnProcessNextEvent(nsIThreadInternal*, bool, unsigned int)|EXCEPTION_ACCESS_VIOLATION_EXEC (74 crashes)
    301% (223/74) vs.   3% (1364/42848) msvcp80.dll
    304% (225/74) vs.   9% (3857/42848) msvcr80.dll
    200% (148/74) vs.   1% (254/42848) atl80.dll
    100% (74/74) vs.   0% (126/42848) TanzanLight.dll
    100% (74/74) vs.   0% (153/42848) riched20.dll
    100% (74/74) vs.   0% (209/42848) RapportGP.dll
    100% (74/74) vs.   0% (214/42848) rooksdol.dll
    100% (74/74) vs.   0% (214/42848) RapportUtil.dll
    100% (74/74) vs.   0% (214/42848) rookscom.dll
    100% (74/74) vs.   1% (240/42848) rooksbas.dll
    100% (74/74) vs.  10% (4123/42848) GdiPlus.dll
    100% (74/74) vs.  18% (7725/42848) oleacc.dll
    100% (74/74) vs.  23% (10041/42848) wtsapi32.dll

Breakdown by module:

100% (74/74) vs.   0% (214/42848) rooksdol.dll
          0% (0/74) vs.   0% (4/42848) 2.115.0.0
        100% (74/74) vs.   0% (205/42848) 2.116.0.0
          0% (0/74) vs.   0% (1/42848) 2.94.0.0
          0% (0/74) vs.   0% (2/42848) 2.97.0.0
          0% (0/74) vs.   0% (2/42848) 2.98.0.0
100% (74/74) vs.   0% (214/42848) RapportUtil.dll
          0% (0/74) vs.   0% (1/42848) 3.5.1105.59
          0% (0/74) vs.   0% (1/42848) 3.5.1108.52
          0% (0/74) vs.   0% (1/42848) 3.5.1108.55
          0% (0/74) vs.   0% (2/42848) 3.5.1108.73
          0% (0/74) vs.   0% (1/42848) 3.5.1201.78
          0% (0/74) vs.   0% (1/42848) 3.5.1201.84
          0% (0/74) vs.   0% (2/42848) 3.5.1201.91
          0% (0/74) vs.   0% (76/42848) 3.5.1201.94
          0% (0/74) vs.   0% (1/42848) 3.5.1201.95
          1% (1/74) vs.   0% (1/42848) 3.5.1201.98
         99% (73/74) vs.   0% (127/42848) 3.5.1205.4
    100% (74/74) vs.   0% (214/42848) rookscom.dll
        100% (74/74) vs.   0% (213/42848) 2.16.0.0
          0% (0/74) vs.   0% (1/42848) 2.9.0.0
    100% (74/74) vs.   1% (240/42848) rooksbas.dll
          0% (0/74) vs.   0% (6/42848) 2.56.0.0
        100% (74/74) vs.   1% (234/42848) 2.57.0.0


Frame 	Module 	Signature 	Source
0 		@0xa491336b 	
1 		@0x7198ffff 	
2 	xul.dll 	nsBaseAppShell::OnProcessNextEvent 	widget/xpwidgets/nsBaseAppShell.cpp:280
3 	xul.dll 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:586
4 	xul.dll 	mozilla::ipc::MessagePump::Run 	ipc/glue/MessagePump.cpp:82
5 	xul.dll 	MessageLoop::RunHandler 	ipc/chromium/src/base/message_loop.cc:201
6 	xul.dll 	MessageLoop::Run 	ipc/chromium/src/base/message_loop.cc:175
7 	xul.dll 	nsBaseAppShell::Run 	widget/xpwidgets/nsBaseAppShell.cpp:163
8 	xul.dll 	nsAppShell::Run 	widget/windows/nsAppShell.cpp:232
9 	xul.dll 	nsAppStartup::Run 	toolkit/components/startup/nsAppStartup.cpp:255
10 	xul.dll 	XREMain::XRE_mainRun 	toolkit/xre/nsAppRunner.cpp:3786
11 	xul.dll 	XREMain::XRE_main 	toolkit/xre/nsAppRunner.cpp:3863
12 	xul.dll 	XRE_main 	toolkit/xre/nsAppRunner.cpp:3939
13 	firefox.exe 	wmain 	toolkit/xre/nsWindowsWMain.cpp:100
14 	firefox.exe 	__tmainCRTStartup 	crtexe.c:552
15 	kernel32.dll 	BaseThreadInitThunk 	
16 	ntdll.dll 	__RtlUserThreadStart 	
17 	ntdll.dll 	_RtlUserThreadStart
RapportUtil.dll belongs to Trusteer Rapport.
Summary: crash in nsBaseAppShell::OnProcessNextEvent → crash in nsBaseAppShell::OnProcessNextEvent with Trusteer Rapport
15.0b6 has 73% of related crashes over the past week, and 14.0.1 has only 15%. Looks like something tickled this crash in our final beta.

Changes in b6: https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?fromchange=5b309ad1a88a&tochange=2099c2bea208

Most of the crashes are happening in the first 15min of use. Can we try to reproduce?
Here are some manual correlations from today:

nsBaseAppShell::OnProcessNextEvent(nsIThreadInternal*, bool, unsigned int)|EXCEPTION_ACCESS_VIOLATION_EXEC (366 crashes)
     19% (68/366) vs.   6% (4051/66577) {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d} (Adblock Plus, https://addons.mozilla.org/addon/1865)
     15% (54/366) vs.   3% (1760/66577) jqs@sun.com (Java Quick Starter, http://java.sun.com/javase/downloads/)
     10% (37/366) vs.   1% (531/66577) {82AF8DCA-6DE9-405D-BD5E-43525BDAD38A}
     13% (48/366) vs.   4% (2746/66577) {CAFEEFAC-0016-0000-0033-ABCDEFFEDCBA}
     11% (42/366) vs.   4% (2385/66577) {b9db16a4-6edc-47ec-a1f4-b86292ed211d} (Video DownloadHelper, https://addons.mozilla.org/addon/3006)
      8% (29/366) vs.   0% (41/66577) {c55f5517-246e-4426-b745-ee25b08eb8b4}
     14% (52/366) vs.   7% (4561/66577) {635abd67-4fe9-1b23-4f01-e679fa7484c1} (Yahoo! Toolbar, https://addons.mozilla.org/addon/2032)
     10% (37/366) vs.   3% (2060/66577) {F53C93F1-07D5-430c-86D4-C9531B27DFAF}
      7% (25/366) vs.   0% (299/66577) en-GB@dictionaries.addons.mozilla.org (British English Dictionary, https://addons.mozilla.org/addon/3366)
      7% (27/366) vs.   1% (674/66577) {BBDA0591-3099-440a-AA10-41764D9DB4DB}
     11% (39/366) vs.   5% (3585/66577) avg@toolbar
      6% (23/366) vs.   1% (816/66577) {D4DD63FA-01E4-46a7-B6B1-EDAB7D6AD389} (Download Statusbar, https://addons.mozilla.org/addon/26)
I'm looking at this report from 15 final:
https://crash-stats.mozilla.com/report/index/745af283-745e-478b-aed9-56a1d2120828

Some interesting things show up: In correlation report there is seemingly strange data:

312% (1141/366) vs.  12% (7743/66577) msvcr80.dll
300% (1099/366) vs.   5% (3380/66577) msvcp80.dll
199% (728/366) vs.   2% (1256/66577) atl80.dll
100% (365/366) vs.   1% (672/66577) RapportGP.dll

But this means that everyone who experienced this crash had multiple copies of the CRT loaded. In this particular crash:

Module|msvcr80.dll|8.0.50727.762|msvcr80.i386.pdb|325121EDEC7C4D62B0DC6FD5CA9080F91|0x74800000|0x7489afff|0
Module|msvcr80.dll|8.0.50727.762|msvcr80.i386.pdb|325121EDEC7C4D62B0DC6FD5CA9080F91|0x74970000|0x74a0afff|0
Module|msvcr80.dll|8.0.50727.762|msvcr80.i386.pdb|325121EDEC7C4D62B0DC6FD5CA9080F91|0x74f20000|0x74fbafff|0

If this data is 100% correct, then it means that the exact same DLL is loaded three times into the address space at different base addresses. It's possible that there is a problem with the minidump collection, though, and this is actually different copies/versions of the CRT being loaded into the address space.

When I load the minidump into MSVC, the stack is very different:
  	a497336b()	
>	xul.dll!nsDOMSVGZoomEvent::nsDOMSVGZoomEvent(aPresContext=, aEvent=)  Line 64	C++

I suspect that we're crashing in the weeds and neither of these stacks (the MSVC one or the breakpad one) are really correct. I'll run the frame-finder script on this minidump.
Attached file Dump lookup results
So I think breakpad got basically the correct stack, we appear to be at

http://hg.mozilla.org/releases/mozilla-release/file/450143d2d810/widget/xpwidgets/nsBaseAppShell.cpp#l298

6D0F62E6  call        nsAppShell::ProcessNextNativeEvent (6D0F63B0h)  

But down the stack that function never shows up. So we're probably seeing stack corruption, but I don't really have any further clues about what could be causing it.
It's #14 top browser crasher in 20.0 and is still correlated with Trusteer Rapport:
     99% (106/107) vs.   1% (201/39209) RapportTanzan20.DLL (3.5.1208.33)
Keywords: topcrash
Version: 15 Branch → 20 Branch
We'll reach out to Trusteer about this issue.
Trusteer confirmed that they released a fix a few days ago. The volume of this crash is already going down and should continue to do so.
(In reply to lsblakk@mozilla.com from comment #9)
> Trusteer confirmed that they released a fix a few days ago.
It seems so based on correlations:
    100% (220/221) vs.   1% (388/74545) RapportTanzan20.DLL
         99% (219/221) vs.   0% (310/74545) 3.5.1208.33
          0% (1/221) vs.   0% (78/74545) 3.5.1208.34
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Whiteboard: [fixed in Trusteer Rapport 3.5.1208.34]
See Also: → 1235257
You need to log in before you can comment on or make changes to this bug.