Crashes at libsystem_pthread.dylib@0x1bb3 (pthread_mutex_unlock) accessing 0x5a5a5a5a5a5a5a5a on Yosemite DP6 and Public Preview 2 using file picker

RESOLVED WORKSFORME

Status

()

Core
Widget: Cocoa
RESOLVED WORKSFORME
4 years ago
4 years ago

People

(Reporter: smichaud, Unassigned)

Tracking

Trunk
All
Mac OS X
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

(crash signature)

Attachments

(1 attachment)

These are almost certainly an Apple bug.  So Cocoa Widgets is (probably) just a placeholder.

For years we've been plagued by Apple file picker bugs, but have never been able to reproduce them.  Many of these were Mac topcrashers, for at least a while.

I can reproduce these!!!  They're currently the #8 Mac topcrasher on the 34 branch.

As best I can tell these happen in all FF versions (I've tested FF 31 and current mozilla-central nightly).

I'll investigate them further when I get the chance (probably next week).
(Reporter)

Comment 2

4 years ago
Forgot to mention that these crashes are on a secondary thread.

But the main thread is definitely in system file picker code.
(Reporter)

Comment 3

4 years ago
Created attachment 8480118 [details]
lldb all-thread stack trace

This was made using today's mozilla-central nightly.
(Reporter)

Updated

4 years ago
Summary: Crashes at libsystem_pthread.dylib@0x1bb3 on Yosemite DP6 and Public Preview 2 using file picker → Crashes at libsystem_pthread.dylib@0x1bb3 (pthread_mutex_unlock) on Yosemite DP6 and Public Preview 2 using file picker
(Reporter)

Updated

4 years ago
Crash Signature: [@ libsystem_pthread.dylib@0x1bb3 ]
(Reporter)

Comment 4

4 years ago
Crash-stats claims these are all null-dereferences.  But they aren't -- none of them are.  Instead they're crashes at address 0x5a5a5a5a5a5a5a5a -- a value jemalloc uses to poison memory that it's released.

I'm pretty sure this means that these crashes are an Apple use-after-free bug.  As far as I know, jemalloc takes over all memory management in the current process -- including for OS code.
(Reporter)

Comment 5

4 years ago
Weirdly, I can't reproduce these crashes using my Mac ASan build from http://people.mozilla.org/~stmichaud/bmo/firefox-asan.dmg.

I know jemalloc likes to re-use the memory from deleted objects as soon as possible (much sooner than happens without jemalloc).  That's probably why.
(Reporter)

Comment 6

4 years ago
I should have mentioned that ASan builds don't use jemalloc.
(Reporter)

Updated

4 years ago
Summary: Crashes at libsystem_pthread.dylib@0x1bb3 (pthread_mutex_unlock) on Yosemite DP6 and Public Preview 2 using file picker → Crashes at libsystem_pthread.dylib@0x1bb3 (pthread_mutex_unlock) accessing 0x5a5a5a5a5a5a5a5a on Yosemite DP6 and Public Preview 2 using file picker
(Reporter)

Comment 7

4 years ago
> libsystem_pthread.dylib`pthread_mutex_unlock + 16:
> -> 0x7fff8a869bb3:  cmpq   $0x4d555458, (%rbx)

RBX here is (0x5a5a5a5a5a5a5a5a + 8) == 0x5a5a5a5a5a5a5a62

You can see this from the raw dump of a Socorro crash stack.
(Reporter)

Comment 8

4 years ago
> I know jemalloc likes to re-use the memory from deleted objects as soon as possible
> (much sooner than happens without jemalloc).

Mike:  Are you familiar with this jemalloc behavior?  Do you know why it happens?  Is there any way to tune it?

Given what I've been able to find out here, I suspect jemalloc has played a role in Apple's file picker crashes going back many years.  I'm not saying this is a jemalloc bug.  But if jemalloc re-uses deleted objects much more quickly than Apple's "standard" memory management, that would tend to uncover many more Apple use-after-free bugs than Apple's own apps do.
Flags: needinfo?(mh+mozilla)
(Reporter)

Comment 9

4 years ago
Since I can reproduce these crashes (and presumably will continue to be able to do so), when I find time I'll track down the specific use-after-free bug (presumably Apple's) that's causing them.  Hopefully that'll be sometime next week.
(Following up comment #8)

Actually it's another, relatively new jemalloc behavior that's most likely to be triggering *these* crashes.  Memory poisoning itself is more likely to uncover use-after-free bugs, since any use-after-free (even one that happens very quickly) will now trigger a crash.
Hit this on Aurora when trying to upload an attachement to Bugzilla: https://crash-stats.mozilla.com/report/index/0f844b89-957c-40cd-ae2d-f67202140829
(Reporter)

Updated

4 years ago
See Also: → bug 1055694
Apple appears to have fixed this particular file picker crash in Yosemite DP7.

And it turns out to have been uninteresting, anyway.  It was a simple, stupid bug that it was presumably very easy for Apple to fix.

jemalloc isn't involved.  Despite my not being able to reproduce these crashes with my Mac ASan build (which doesn't use jemalloc), I later found out that (in non-ASan builds) these crashes happen with or without jemalloc.

And, though these crashes happen on a secondary thread, there are no race conditions involved.

All ..._block_invoke methods are called with a single parameter -- an object that inherits from NSBlock.  The one involved here (AppKit`__42-[_NSScrollingConcurrentVBLMonitor resume]_block_invoke) gets called multiple times with the same NSBlock object, and then releases it the last time through.  But it releases it too early, before it's done with it.

So this bug is a very simple, straightforward use-after-free.
(Reporter)

Updated

4 years ago
See Also: bug 1055694
(Reporter)

Updated

4 years ago
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → WORKSFORME
(Reporter)

Updated

4 years ago
Flags: needinfo?(mh+mozilla)
> All ..._block_invoke methods are called with a single parameter -- an object that
> inherits from NSBlock.

I should have said that they all get called with *at least* a single NSBlock parameter (the first one).  There are sometimes also other parameters.
You need to log in before you can comment on or make changes to this bug.