1467230 - [10.14] alloc-related crash in libobjc.A.dylib@0x6e1d

Status: RESOLVED WORKSFORME

(Core :: General, defect)

Description

[Marcia Knous [:marcia]]

This bug was filed from the Socorro interface and is
report bp-1f6f6ed1-c7b1-4e43-8bb8-e1d8b0180606.
=============================================================

Seen while looking at crash stats - all of these crashes occur on the latest 10.14 Mojave beta: https://bit.ly/2JhPm7e

Marking as security sensitive since they may be UAFs based on the crash signature.

Comments:
*It looks like Firefox and the latest macOS beta don’t like each other. 
*Not working with macOS 10.14 Mojave 


Top 10 frames of crashing thread:

0 libobjc.A.dylib libobjc.A.dylib@0x6e1d 
1 libdispatch.dylib libdispatch.dylib@0x80ce 
2 libdispatch.dylib libdispatch.dylib@0x14d1 
3 libdispatch.dylib libdispatch.dylib@0x14199 
4 libdispatch.dylib libdispatch.dylib@0x14c51 
5 libdispatch.dylib libdispatch.dylib@0x1b339 
6 libsystem_pthread.dylib libsystem_pthread.dylib@0x28e8 
7 libsystem_pthread.dylib libsystem_pthread.dylib@0x273c 
8 libsystem_pthread.dylib libsystem_pthread.dylib@0x273c 
9 libmozglue.dylib arena_t::AllocRun memory/build/mozjemalloc.cpp:2558

=============================================================

Comment 1

[Daniel Veditz [:dveditz]]

Did something change in the Mac system allocators in 10.14? Crashing with our UAF marker deep into system routines but under our allocator routines (at least in the crashes I checked).

Since this is Mac glandium probably isn't the right person to investigate, but I'm not sure who else would look at the platform nitty-gritty on mac.

Comment 2

[Mike Hommey [:glandium]]

So, one problem is that we don't have the symbols for those system libraries, so the traces past the first frame could be completely wrong, and the allocator might not be involved at all. I wouldn't be entirely surprised if there's an UAF in some system library... Anyways, I'd say the first thing we should do here is grab the 10.14 system libraries and add them to soccoro. Ted, can you do that?

Once that's done, we can either reprocess the crash reports, or just wait for more to come in, and they will have more accurate traces.

Comment 3

[Marcia Knous [:marcia]]

There is an 85% correlation to One Password: (85.71% in signature vs 00.92% overall) Addon "[e-mail]" = true.

Comment 4

[Alex Gaynor [:Alex_Gaynor]]

Could that just be because people on 10.14 are way more likely to also use One Password?

Comment 5

[Marcia Knous [:marcia]]

(In reply to Alex Gaynor [:Alex_Gaynor] from comment #4)
> Could that just be because people on 10.14 are way more likely to also use
> One Password?

Possibly. But right now that is the strongest addon correlation we have. We still need the symbols for the system libraries which Mike calls out in Comment 2.

Comment 6

[Marcia Knous [:marcia]]

Some of the recent comments indicate that this is also happening in safe mode.

Comment 7

[Marcia Knous [:marcia]]

Apple just came out with a new 10.14 seed yesterday (10.14.0 18A314h). It looks to me as if all the crash reports so far for the two signatures listed in this bug are running version 10.14.0 18A293u. I will keep an eye out for new reports that are running the new seed.

Comment 8

[(not currently active) Ted Mielczarek]

I don't have a good process for getting symbols for a full macOS release currently. If someone with a 10.14 install handy wants to run the script here we could get that uploaded to symbols.mo:
https://github.com/luser/breakpad-scrape-system-symbols/

(I'm on PTO next week so I won't get to it.)

Comment 9

[Marcia Knous [:marcia]]

(In reply to (PTO June 23rd-30th) Ted Mielczarek [:ted.mielczarek] from comment #8)
> I don't have a good process for getting symbols for a full macOS release
> currently. If someone with a 10.14 install handy wants to run the script
> here we could get that uploaded to symbols.mo:
> https://github.com/luser/breakpad-scrape-system-symbols/
> 
> (I'm on PTO next week so I won't get to it.)

I tried running the script today and got an error. I can try to debug with you when you get back from PTO.

Comment 10

[Marcia Knous [:marcia]]

Adding a new signature that I spotted during nightly crash triage. This report comes from 10.14.0 18A314h, which is the latest seed available.

Comment 11

[Marcia Knous [:marcia]]

While doing my nightly crash triage, I do see additional signatures such as [@ libobjc.A.dylib@0x906c] - https://crash-stats.mozilla.com/report/index/42ea2332-9bc1-4f4a-a6d9-ee50f0180715- that appear to not have the alloc part in the crash stack but still appear to be UAFs.

I have been working with Ted offline to try to get symbols - I have run into some errors that have prevented me from getting them.

Comment 12

[Marcia Knous [:marcia]]

Still seeing some UAF signatures, the most recent is https://crash-stats.mozilla.com/report/index/59c9f35e-5637-4e51-aa78-b7dc70180726 although I don't know if this is the same bug as this one. Guess we will no more when we have symbols?

Comment 13

[Mike Hommey [:glandium]]

FWIW, I've been running Firefox on the latest beta of Mojave without problems for the past few hours, so we're out of the woods as far as straight out incompatibilities are concerned.
I tried scraping the system symbols, but people.mozilla.org not being there anymore, ted's prebuilt dump_syms is not there, and somehow I can't get it to build from the breakpad source. I guess I'll have to try to do that from the firefox source.

Comment 14

[Marcia Knous [:marcia]]

We now have symbols for 10.14, see Bug 1480101. The signatures in this bug are all from older seeds, mostly 10.14.0 18A293u and a few others. I don't see any crashes in these signatures in the most recent seed, 18A377a. But I guess the signature could have changed. I just filed Bug 1487722 for a possible UAF in the most recent seed.

Comment 15

[Marcia Knous [:marcia]]

Although the first 5 signatures were only happening in earlier versions of Mojave, I noticed that in the final shipped version (10.14.0 18A391) there are a still a handful of crashes in [@ libobjc.A.dylib@0x9d5c] that have that signature (Firefox and Thunderbird): https://bit.ly/2Iy3IwG. It appears 63 is affected, so marking that as well. 

https://crash-stats.mozilla.com/report/index/bd263780-8472-49d3-8c3a-66cde0180928 is one example from 62.0.2:

14 	XUL 	-[ChildView preRender:] 	widget/cocoa/nsChildView.mm:3419
15 	XUL 	nsChildView::PreRender(mozilla::widget::WidgetRenderingContext*) 	widget/cocoa/nsChildView.mm:2147
16 	XUL 	mozilla::layers::LayerManagerComposite::Render(mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&) 	gfx/layers/composite/LayerManagerComposite.cpp:892
17 	XUL 	mozilla::net::nsHttpDigestAuth::GenerateCredentials(nsIHttpAuthenticableChannel*, char const*, bool, char16_t const*, char16_t const*, char16_t const*, nsISupports**, nsISupports**, unsigned int*, char**)::hexChar 	
18 	XUL 	mozilla::layers::LayerManagerComposite::UpdateAndRender() 	gfx/layers/composite/LayerManagerComposite.cpp:534
19 	XUL 	mozilla::layers::LayerManagerComposite::EndTransaction(mozilla::TimeStamp const&, mozilla::layers::LayerManager::EndTransactionFlags) 	gfx/layers/composite/LayerManagerComposite.cpp:464
20 	XUL 	mozilla::layers::CompositorBridgeParent::CompositeToTarget(mozilla::gfx::DrawTarget*, mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const*) 	gfx/layers/ipc/CompositorBridgeParent.cpp:1067
21 	XUL 	mozilla::layers::CompositorVsyncScheduler::Composite(mozilla::TimeStamp) 	gfx/layers/ipc/CompositorVsyncScheduler.cpp:243
22 	XUL 	mozilla::detail::RunnableMethodImpl<mozilla::layers::CompositorVsyncScheduler*, void (mozilla::layers::CompositorVsyncScheduler::*)(mozilla::TimeStamp), true, (mozilla::RunnableKind)1, mozilla::TimeStamp>::Run() 	xpcom/threads/nsThreadUtils.h:1165
23 	XUL 	MessageLoop::DeferOrRunPendingTask(MessageLoop::PendingTask&&) 	ipc/chromium/src/base/message_loop.cc:451
24 	XUL 	MessageLoop::DoWork() 	ipc/chromium/src/base/message_loop.cc:534
25 	XUL 	base::MessagePumpDefault::Run(base::MessagePump::Delegate*) 	ipc/chromium/src/base/message_pump_default.cc:38
26 	XUL 	MessageLoop::Run() 	ipc/chromium/src/base/message_loop.cc:325
27 	XUL 	base::Thread::ThreadMain() 	ipc/chromium/src/base/thread.cc:181

Comment 16

[Daniel Veditz [:dveditz]]

The most recent signature you added, @0x9d5c is nowhere near the others. Maybe the others are related signature drifts in various versions, but this one clearly is something else. Not sure it's helpful to lump these together. The stack is completely different, too. The original crash was in allocation code, this one is in widget resizing.

Can you please put the @0x9d5c signature in a separate new bug. The original bug/signatures look like they're going away.

Comment 17

[Marcia Knous [:marcia]]

(In reply to Daniel Veditz [:dveditz] from comment #16)
> The most recent signature you added, @0x9d5c is nowhere near the others.
> Maybe the others are related signature drifts in various versions, but this
> one clearly is something else. Not sure it's helpful to lump these together.
> The stack is completely different, too. The original crash was in allocation
> code, this one is in widget resizing.
> 
> Can you please put the @0x9d5c signature in a separate new bug. The original
> bug/signatures look like they're going away.

Hello Dan - Per your request I removed that signature and will file a separate bug.

Comment 18

[Alex Gaynor [:Alex_Gaynor]]

Crash has not been seen since august.