Closed Bug 680954 Opened 13 years ago Closed 12 years ago

[10.5] Firefox crash __addAltHandler2

Categories

(Firefox :: General, defect)

Product:
Firefox
Component:
General
Version:
7 Branch
Platform:
x86
macOS
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

()

Status:
RESOLVED WORKSFORME
Tracking Flags:
Tracking Status
firefox16 --- affected
firefox17 --- unaffected

People

(Reporter: marcia, Unassigned)

Details

(Keywords: crash, topcrash)

Crash Data

This bug was filed from the Socorro interface and is 
report bp-f1b89c71-b5ec-416a-82c8-ac70e2110821 .
============================================================= 

Seen while looking at crash reports. https://crash-stats.mozilla.com/report/list?signature=__addAltHandler2 links to the crashes which happen across versions. High concentration of 10.5.x crashes.

Two comments mention "The browser had been acting extremely slow for several minutes for the crash"

Frame 	Module 	Signature [Expand] 	Source
0 	CoreFoundation 	__addAltHandler2 	
1 	CoreFoundation 	_CFDoExceptionOperation 	
2 	Foundation 	_NSAddExceptionHandlerForLock 	
3 	AppKit 	_NSAppKitLock 	
4 	AppKit 	-[NSApplication _findWindowUsingCache:] 	
5 	AppKit 	-[NSEvent window] 	
6 	AppKit 	-[NSEvent _cgsEventRecord] 	
7 	AppKit 	-[NSEvent _eventRefInternal] 	
8 	AppKit 	-[NSEvent _postAtStart:] 	
9 	XUL 	nsAppShell::ProcessGeckoEvents 	widget/src/cocoa/nsAppShell.mm:439
10 	CoreFoundation 	__CFRunLoopDoSources0 	
11 	CoreFoundation 	__CFRunLoopRun 	
12 	CoreFoundation 	CFRunLoopRunSpecific 	
13 	CoreFoundation 	CFRunLoopRunInMode 	
14 	HIToolbox 	RunCurrentEventLoopInMode 	
15 	HIToolbox 	ReceiveNextEventCommon 	
16 	HIToolbox 	BlockUntilNextEventMatchingListInMode 	
17 	AppKit 	_DPSNextEvent 	
18 	AppKit 	-[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] 	
19 	XUL 	nsAppShell::ProcessNextNativeEvent 	widget/src/cocoa/nsAppShell.mm:675
20 	XUL 	nsBaseAppShell::OnProcessNextEvent 	widget/src/xpwidgets/nsBaseAppShell.cpp:171
21 	XUL 	nsAppShell::OnProcessNextEvent 	widget/src/cocoa/nsAppShell.mm:856
22 	XUL 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:582
23 	XUL 	NS_ProcessPendingEvents_P 	obj-firefox/i386/xpcom/build/nsThreadUtils.cpp:195
24 	XUL 	nsBaseAppShell::NativeEventCallback 	widget/src/xpwidgets/nsBaseAppShell.cpp:130
25 	XUL 	nsAppShell::ProcessGeckoEvents 	widget/src/cocoa/nsAppShell.mm:422
26 	CoreFoundation 	__CFRunLoopDoSources0 	
27 	CoreFoundation 	__CFRunLoopRun 	
28 	CoreFoundation 	CFRunLoopRunSpecific 	
29 	CoreFoundation 	CFRunLoopRunInMode 	
30 	HIToolbox 	RunCurrentEventLoopInMode 	
31 	HIToolbox 	ReceiveNextEventCommon 	
32 	HIToolbox 	BlockUntilNextEventMatchingListInMode 	
33 	AppKit 	_DPSNextEvent 	
34 	AppKit 	-[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] 	
35 	AppKit 	-[NSApplication run] 	
36 	XUL 	nsAppShell::Run 	widget/src/cocoa/nsAppShell.mm:769
37 	XUL 	nsAppStartup::Run 	toolkit/components/startup/nsAppStartup.cpp:222
38 	XUL 	XRE_main 	toolkit/xre/nsAppRunner.cpp:3686
39 	firefox-bin 	main 	browser/app/nsBrowserApp.cpp:158
40 	firefox-bin 	firefox-bin@0xa05 	
41
It's #3 top crasher in 7.0.1 on Mac OS X.

It happens almost only on Mac OS X 10.5 with Flash Player:
     98% (53/54) vs.  27% (579/2128) FlashPlayer-10.4-10.5
          2% (1/54) vs.   0% (7/2128) 48F84FE6AB8341F943264EE657D564330
          4% (2/54) vs.   1% (15/2128) 618100E683D0F650E7AB834638DB5F6A0
         20% (11/54) vs.   6% (123/2128) 9E0D5FDCE905F569CBE364E85AC98BFC0
          4% (2/54) vs.   3% (71/2128) 9E952D5BE0C7A400FF8D0D280BE6E3990
         37% (20/54) vs.  10% (212/2128) AB20F8CBD0050FDF33413B314A53F9150
          4% (2/54) vs.   1% (18/2128) CCAF6CF191CA2C8EB8ED19375014F2130
          4% (2/54) vs.   0% (4/2128) CDB00B650C699A1DBD9350A6EF4A36CA0
          7% (4/54) vs.   2% (38/2128) ED71BBFA041EF60C49672DDA0CC6569C0
         17% (9/54) vs.   3% (63/2128) FFCFA8DE6C45E84EB674EC324644884A0
Keywords: topcrash
This is still a valid top crash. Sits at #2 spot for all mac crashes on 8.0.1.
Adding Steven to the bug. This shows up in Firefox 9b4 but in pretty small volume - 38 crashes. Based on the correlation, perhaps something is happening when the browser is interacting with Flash on 10.5?
fyi, Flash version support for OSX 10.5 stopped at 10.3.  the latest version is 10,3,183,10.
Stacks with this signature have been around for a long time, across
all versions of Firefox and of OS X.  But it's only in the last few
months that we've started to see so many of them on OS X 10.5.8.
(I'll try to get a timeline for this in a later comment).

These crashes occur deep in system code.  But if I'm interpreting
Breakpad's raw dump format correctly, they're all the result of the
failure of a call to malloc() (on OS X 10.5.8) or calloc() (on OS X
10.6.8) inside the __addAltHandler2() function.  (More, too, on this
in a later comment.)

This is extremely puzzling.  It's very unlikely	that ordinary memory
pressure (i.e. running out of memory) is responsible for these
failures.  Otherwise they'd be just one	part of	a whole	cluster	of
"out of memory" errors,	which I'm pretty sure we haven't seen.

If we'd enabled jemalloc on OS X 10.5 in FF 8.0.1, that'd be my first
suspect.  But we haven't.

Which leads me to wonder if this is some kind of OS bug.  And also to
wonder if Flash has some part in it, perhaps by (somehow) intercepting
calls to malloc() and calloc().  (Once again, more on this in a later
comment.)
In the raw dumps of this bug's Breakpad crash stacks on OS X 10.5.8,
the top line of the crash stack is represented as follows:

0|0|CoreFoundation|__addAltHandler2|||0xf6

I take this to mean that these crashes (all of them NULL dereferences)
take place at offset 0xf6 (decimal 246) from the beginning of the
__addAltHandler2() function.  From the assembly code at that offset,
you can see that a call to malloc() that returns NULL will cause a
NULL derefence at that offset:

0x934b13f8 <__addAltHandler2+232>:      movl   $0x9c,(%esp)
0x934b13ff <__addAltHandler2+239>:      call   0xa0a46588 <dyld_stub_malloc>
0x934b1404 <__addAltHandler2+244>:      mov    %eax,%edx
0x934b1406 <__addAltHandler2+246>:      movl   $0xc,(%eax)

The few examples of this crash on OS X 10.6.8 happen at a different
offset from the beginning of __addAltHander2():

0|0|CoreFoundation|__addAltHandler2|||0x12a

Hexadecimal 0x12a translates into decimal 298.  But __addAltHandler2()
is slightly longer on OS X 10.6.8, and its assembly code shows that
these crashes take place as the result of a call to calloc() that
returns NULL:

0x03c5ee54 <__addAltHandler2+276>:      movl   $0x9c,0x4(%esp)
0x03c5ee5c <__addAltHandler2+284>:      movl   $0x1,(%esp)
0x03c5ee63 <__addAltHandler2+291>:      call   0x3d29758 <dyld_stub_calloc>
0x03c5ee68 <__addAltHandler2+296>:      mov    %eax,%ebx
0x03c5ee6a <__addAltHandler2+298>:      movl   $0xc,(%ebx)
On both OS X 10.5.8 and 10.6.8, the current version of the OS-specific
Flash Player "PlugIn" has the following string:

[mem] gc[%d] %s allocator:   %d%% efficiency %d bytes (%d kb) in use out of %d bytes (%d kb)

This seems to imply that Flash has a way of measuring allocator
"efficiency", and some kind of garbage collection.
Another thing to note (though many of you will already have realized it):

Even on OS X 10.6.8 and 10.7.X, these crashes only happen in 32-bit mode.
Yet another thing to note:  The allocations that are failing are very small -- 0x9c bytes (156 decimal).
There were 543 crashes at __addAltHandler2 in all FF versions in the
last week.  The number slowly falls as you go further back.  But the
most dramatic change that I can find is	between the week ending
2011/08/28 (324) and the one ending 2011/08/21 (206).

This is just after FF 6.0 was released.  I don't know how it
corresponds to the Flash release schedule.
(In reply to Steven Michaud from comment #10)
> This is just after FF 6.0 was released.  I don't know how it
> corresponds to the Flash release schedule.
Flash 10.3 was released in May 2011 and Flash 11.0 in October 2011 (see http://en.wikipedia.org/wiki/Adobe_Flash_Player#Release_history).
There was a security fix in Flash 10.3.183.5 released on August 12th (see http://www.adobe.com/support/security/bulletins/apsb11-21.html).
(Following up comment #5)

> It's very unlikely that ordinary memory pressure (i.e. running out
> of memory) is responsible for these failures.  Otherwise they'd be
> just one part of a whole cluster of "out of memory" errors, which
> I'm pretty sure we haven't seen.

And if these were ordinary out-of-memory symptoms, you'd think high
uptimes would be heavily overrepresented in the crash stats.  But as
best I can tell (by eyeballing them) the uptimes are pretty evenly
distributed.

Anyone know what the "uptime" unit is?  Is it seconds?
(In reply to Steven Michaud from comment #12)
> Anyone know what the "uptime" unit is?  Is it seconds?
Yes.

It's currently #9 top crasher in 10.0 and #16 in 11.0b1 on Mac OS X.
This still sitting in the top 5 for Mac crashes on release. At #5 on 10.0.2 with 300 crashes a week.
99% of crashes happen on Mac OS X 10.5.
Summary: Firefox crash __addAltHandler2 → [10.5] Firefox crash __addAltHandler2
Let's close it as WFM now that Mac OS X 10.5 is no longer supported in Firefox 17.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.