1415760 - Crash in mozilla::dom::TimeoutExecutor::ScheduleDelayed

Status: RESOLVED INCOMPLETE

(Core :: DOM: Core & HTML, defect, P1)

Description

[Randell Jesup [:jesup] (needinfo me)]

This bug was filed from the Socorro interface and is
report bp-121ae250-9daf-454b-ae84-18cc50171103.
=============================================================

Top 10 frames of crashing thread:

0 xul.dll mozilla::dom::TimeoutExecutor::ScheduleDelayed dom/base/TimeoutExecutor.cpp:58
1 xul.dll icu_59::DecimalFormatSymbols::DecimalFormatSymbols intl/icu/source/i18n/dcfmtsym.cpp:112
2 xul.dll icu_59::NumberFormat::makeInstance intl/icu/source/i18n/numfmt.cpp:1398
3 xul.dll icu_59::NumberFormat::internalCreateInstance intl/icu/source/i18n/numfmt.cpp:1049
4 xul.dll icu_59::LocaleCacheKey<icu_59::SharedNumberFormat>::createObject intl/icu/source/i18n/numfmt.cpp:1260
5 xul.dll icu_59::UnifiedCache::_get intl/icu/source/common/unifiedcache.cpp:409
6 xul.dll icu_59::UnifiedCache::get<icu_59::SharedNumberFormat> intl/icu/source/common/unifiedcache.h:234
7 xul.dll icu_59::UnifiedCache::getByLocale<icu_59::SharedNumberFormat> intl/icu/source/common/unifiedcache.h:274
8 xul.dll icu_59::NumberFormat::createSharedInstance intl/icu/source/i18n/numfmt.cpp:1285
9 xul.dll icu_59::NumberFormat::createInstance intl/icu/source/i18n/numfmt.cpp:1057

=============================================================

UAF crash.... spike in 57b13 it looks like, but there are older crashes going back to at least 55.

The stacks (see above) for 57 are kinda crazy; the older stacks seem sensible.

Comment 1

[Ben Kelly [:bkelly, not reviewing]]

The last frame of this stack looks bogus to me.  Its all ICU and then a single DOM TimeoutExecutor frame.  There is no way that ICU is calling that code.  All the stacks I looked at have this pattern.  They are also all from a single installation.

If you have a crash, with a different looking stack, can you please link it here?

Comment 2

[Boris Zbarsky [:bzbarsky]]

Like rjesup said, the 57 stacks look bogus. But looking at https://crash-stats.mozilla.com/signature/?signature=mozilla%3A%3Adom%3A%3ATimeoutExecutor%3A%3AScheduleDelayed&date=%3E%3D2017-11-02T01%3A05%3A56.000Z&date=%3C2017-11-09T00%3A05%3A56.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=install_time&_sort=-date&page=1#reports the very first 56 stack looks like something that could happen: https://crash-stats.mozilla.com/report/index/d3af8337-dfa2-4b80-bcc6-c9a9c0171105

Comment 3

[Randell Jesup [:jesup] (needinfo me)]

Ted: thoughts on the 57 stacks like the one above?  Note that there were older, saner (though rare) crashes.

Comment 4

[Ben Kelly [:bkelly, not reviewing]]

Thanks.  It seems the crash in comment 2 is failing on the first member write after nsTimerImpl calls into the TimeoutExecutor.  My guess is somehow nsTimerImpl is somehow calling Notify() on an invalid pointer reference or after dropping its reference somehow.  I don't see how the TimeoutExecutor code could crash in this stack on a live object.

Sorry, I'm not sure I'm going to have any time to look at this further since I'm at TPAC this week.  Maybe Nathan can help with nsITimer stuff?

Comment 5

[Andrew Overholt [:overholt]]

Andreas may also have some insights as he's looking at a similar crash.

Comment 6

[Andreas Farre [:farre]]

I haven't look that far into this, but it seems to be a timer issue at first glance. Will look some more, but help from Nathan is probably a good bet.

Comment 7

[Andreas Farre [:farre]]

So the call stack for this is fairly varied, I see TimeoutExecutor::ScheduleDelayed originating from nsGlobalWindow::SetTimeoutOrInterval, nsTimerImpl::Fire, TimeoutExecutor::Run, but when searching for mozilla::dom::TimeoutExecutor::ScheduleDelayed[1] I also see (fewer though) what appears to be mutex related issues, also in TimeoutExecutor::ScheduleDelayed.

Could that be the issue for all of these cases?

[1] https://crash-stats.mozilla.com/search/?signature=~mozilla%3A%3Adom%3A%3ATimeoutExecutor%3A%3AScheduleDelayed&date=%3E%3D2017-01-01T16%3A35%3A00.000Z&date=%3C2017-11-22T16%3A35%3A40.000Z&_sort=-date&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature

Comment 8

[Ben Kelly [:bkelly, not reviewing]]

I'm not sure what you mean about "mutex related issues".  The stacks I looked at seemed like trying to run the mutex code on a bad address or uninitialized memory.  This again suggests the nsTimer is firing on a dead object.

Comment 9

[Andreas Farre [:farre]]

So I meant races, but as you say, those seem to be all coming from the timer firing on a bad object.

What I really was trying to understand was why e.g. https://crash-stats.mozilla.com/report/index/f99881f3-fd24-4832-81df-9ff940171120 happens and how that ties in with the other stack traces. Here we're coming from a SetTimeoutOrInterval, and the only thing that I can understand it as is mOwner in TimeoutExecutor being garbage.

Comment 10

[Andrew Overholt [:overholt]]

Andreas and I spoke. He's still investigating but it's really hard to get anywhere here. I'll mark it as P1 given the UAF aspect but I'm not sure how soon we can expect a breakthrough here.

Comment 11

[Andreas Farre [:farre]]

I wonder if Bug 1421009 is the reason behind this.

Comment 12

[Frederik Braun [:freddy]]

(In reply to Andreas Farre [:farre] from comment #11)
> I wonder if Bug 1421009 is the reason behind this.

Do you think there could be some diagnostic asserts we could add to find out?

Comment 13

[Andreas Farre [:farre]]

I don't think so, at least not after fixing Bug 1421009.

Comment 14

[Nathan Froyd [:froydnj]]

crash-stats currently only gives me one hit for the crash signature in this bug:

https://crash-stats.mozilla.com/signature/?signature=mozilla%3A%3Adom%3A%3ATimeoutExecutor%3A%3AScheduleDelayed&date=%3E%3D2017-10-29T12%3A16%3A14.000Z&date=%3C2018-01-29T10%3A16%3A14.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=install_time&_sort=-date&page=1#reports

https://crash-stats.mozilla.com/report/index/789be3aa-eb54-4ec0-9a94-f750f0171231

which is just indexing off a null pointer.  The buildid is a couple of weeks after Andreas's fix in bug 1421009.

I'm not sure what else to do here, unless there's something I'm missing?

Comment 15

[Randell Jesup [:jesup] (needinfo me)]

I fiddled and found around 10, all nullptr writes to 0x24 in 57.0.x

Perhaps the signature changed?  dumping crash-stats here doesn't help

Comment 16

[Andrew Overholt [:overholt]]

If there's nothing we can do here, I'm tempted to resolve this as INCOMPLETE. Especially given the incredibly low volume and lack of smoking guns in the code. What do people think?

Comment 17

[Frederik Braun [:freddy]]

(In reply to Andrew Overholt [:overholt] from comment #16)
> If there's nothing we can do here, I'm tempted to resolve this as
> INCOMPLETE. Especially given the incredibly low volume and lack of smoking
> guns in the code. What do people think?

WDYT?

Comment 18

[Daniel Veditz [:dveditz]]

There'a low volume of crashes, but it continues on in 58 and 59. About evenly split between those two and evenly split between Desktop and Fennec. There are _WRITE and _EXEC violations mixed in with the _READ ones.