1359840 - Crash in shutdownhang | NtWaitForAlertByThreadId | RtlSleepConditionVariableCS | SleepConditionVariableCS

Status: RESOLVED INCOMPLETE

(Core :: General, defect)

Description

[Marcia Knous [:marcia]]

This bug was filed from the Socorro interface and is 
report bp-d536f50d-df7f-4c00-b5cd-c39682170310.
=============================================================

Probably not in right bucket, maybe nathan can help? 

Seen while looking at nightly crash stats. Right now on nightly this is #6 browser crash overall: http://bit.ly/2qezlRt and has risen a bit in the last week

Affects other branches as well and has been hanging around.

Comment 1

[Nathan Froyd [:froydnj]]

Shutdown hangs during JS GC can't be good.  Jon, does this just mean we have a bunch of garbage to collect, or is this an actual bug in the GC?

Comment 2

[Andrew McCreight [:mccr8]]

For some reason I don't understand, we have to do this shutdown GC. It happens after we've torn everything down, so there's a lot to collect. It does seem bad if it is taking a minute to do, so often.

Comment 3

[Jon Coppeard (:jonco)]

Most of the crashes with this signature don't have GC on the stack.  I think we need a separate bug to the GC ones.

Comment 4

[Andrew McCreight [:mccr8]]

I did a search, and out of the 514 that occurred in the last week, JS_GC was in the proto signature of 313 of them (60% of the total). So, shutdown GCs are a major issue for this signature.

Comment 5

[Jon Coppeard (:jonco)]

Created bug 1359863 for shutdown hangs during GC.

Comment 6

[Jon Coppeard (:jonco)]

(In reply to Andrew McCreight [:mccr8] from comment #4)
Oh right, it wasn't in the first five I looked at so I guessed the proportion was lower than that.

Comment 7

[Marcia Knous [:marcia]]

Today I found a similar signature [@ shutdownhang | NtWaitForKeyedEvent | RtlSleepConditionVariableCS | SleepConditionVariableCS], but wasn't sure whether it belongs in this bug or needs a separate bug.

Comment 8

[Ryan VanderMeulen [:RyanVM]]

This signature is currently showing up as the #4 nightly topcrasher. Not sure if there's another bug tracking it since this is a bit old?

Comment 9

[Marcia Knous [:marcia]]

I haven't seen another bug tracking it. A comments from a user:

froze while trying to shift back through the tabs open to find the one i wanted. stayed on a page with the top left frozen spinning page wheel as if reloading. mouse stayed on hand pointer but no response from page/ hand never changed shape over buttons and so on. I stopped trying to scroll back once this began, and ran these tests. also - page did successfully minimize and maximize, but it wouldn't reload and wouldn't complete th process it had frozen on. Closed from button on Windows taskbar 

ni on :jonco so the JS team can triage this bug and see if we can do anything about it as it is continuing to happen on nightly.

Comment 10

[Bill Gianopoulos [:WG9s]]

This happens for me when I do the restart to apply an update.

Comment 11

[Wayne Mery (:wsmwk)]

#1 crash for Thunderbird nightly

Comment 12

[Trevor Skywalker]

[Tracking Requested - why for this release]:

This is the #4 top crasher for Nightly.

Top Crashers for Firefox 57.0a1
Top 50 Crashing Signatures. 7 days ago through 34 minutes ago. 

#4 	
2.92% 	
-0.46% 	
shutdownhang | NtWaitForAlertByThreadId | RtlSleepConditionVariableCS | SleepConditionVariableCS
	275 	275 	0 	0 	241 	0 	2015-04-01

Comment 13

[Gerry Chang [:gchang]]

Track 56+/57+ as top crash.

Comment 14

[Liz Henry (:lizzard) (relman/hg->git project)]

This is by far the top crash in the 56 RC1 from last week. Is there anything possibly actionable here?

Comment 15

[Jon Coppeard (:jonco)]

This is an overly-broad signature.  Less that 1% of the crashes under this signature in the last week are related to GC.

The top one (26%) is:

NtWaitForAlertByThreadId | RtlSleepConditionVariableCS | SleepConditionVariableCS | mozilla::detail::ConditionVariableImpl::wait | mozilla::CondVar::Wait | nsEventQueue::GetEvent | nsThread::nsChainedEventQueue::GetEvent | nsThread::GetEvent | nsThread::ProcessNextEvent | NS_ProcessNextEvent | mozilla::dom::workers::RuntimeService::Cleanup | mozilla::dom::workers::RuntimeService::Observe | nsObserverList::NotifyObservers | nsObserverService::NotifyObservers | mozilla::ShutdownXPCOM | ScopedXPCOMStartup::~ScopedXPCOMStartup 

This crash and others like it are covered by signatures in bug 1368983 and bug 1267692.

Comment 16

[Andrew McCreight [:mccr8]]

Shutdown hangs are just always a top crash, AFAICT. The signature here actually buckets together a lot of issues. Maybe bug 1402037 will help with that.

Looking over the signatures from beta:
- About 40% are hangs while spinning the event loop while shutting down DOM workers:
  bp-335ca6ec-1519-4496-ae06-bb3dc0170922
I'll file an issue for that, but I'm not sure what we can do.

- About 22% of the hangs are from spinning the event loop in CompositorThreadHolder::Shutdown. There's a bug somewhere on a related issue, I'll see if I can find it. bp-450f7d25-8d03-4c27-bf5b-041160170922

- About 6% are related to XHR. I assume this is related to sync XHR while we're shutting down, but that's just a guess. bp-b0da69b2-7c58-4a92-a888-6d2a60170922

I also see a few other places where we're spinning the event loop during shutdown, including the QuotaManager (about 6% of these hangs) and nsHttpConnectionManager (about 5% of these hangs). Anything else seems to be less than about 1%.

Comment 17

[Andrew McCreight [:mccr8]]

> I'll file an issue for that, but I'm not sure what we can do.
Ah, looks like Jon found an existing issue for it, bug 1267692.

Comment 18

[Andrew McCreight [:mccr8]]

(In reply to Andrew McCreight [:mccr8] from comment #16)
> - About 22% of the hangs are from spinning the event loop in
> CompositorThreadHolder::Shutdown. There's a bug somewhere on a related
> issue, I'll see if I can find it. bp-450f7d25-8d03-4c27-bf5b-041160170922
Bug 1389021 is what I was thinking of. Though it is marked fixed. I guess that didn't fix everything.

Comment 19

[Liz Henry (:lizzard) (relman/hg->git project)]

I'm not seeing this specific signature right now in crash-stats, though there are other shutdown hangs still very active (like in bug 1405290).

Comment 20

[Wayne Mery (:wsmwk)]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #11)
> #1 crash for Thunderbird nightly

Too funny - For Thunderbird on Aug 9, the day I made the above comment, TB nightly crash rate dropped, apparently with the introduction of 57.0a1. The signature continues only in small amounts for Thunderbird.


(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #19)
> I'm not seeing this specific signature right now in crash-stats, though
> there are other shutdown hangs still very active (like in bug 1405290).

Indeed, because of socorro processing change via bug 1402037.  The signature ends abruptly after 2017-09-22 for all versions of firefox.

Comment 21

[Wayne Mery (:wsmwk)]

For firefox, the signature is still gone

For Thunderbird, the crashes have changed to signatures such as 
 shutdownhang | arena_t::MallocSmall | nsCOMPtr_base::assign_with_AddRef | mozilla::EventQueue::PutEvent 
for example bp-dec2b4db-d2e0-48ad-a8fb-3fff40171228