1790983 - Crash in [@ shutdownhang | mozilla::OffTheBooksCondVar::Wait]

Status: RESOLVED FIXED

(Core :: XPCOM, defect)

Description

[Wayne Mery (:wsmwk)]

Possibly during/after an update of daily build. The crash reporter was there when I logged on to the computer.

Crash report: https://crash-stats.mozilla.org/report/index/058c0c51-c3dc-4bdb-8a3d-9dd2a0220915

MOZ_CRASH Reason: Shutdown hanging at step XPCOMShutdownThreads. Something is blocking the main-thread.

Top 10 frames of crashing thread:

0 ntdll.dll NtWaitForAlertByThreadId 
1 ntdll.dll RtlSleepConditionVariableSRW 
2 KERNELBASE.dll SleepConditionVariableSRW 
3 mozglue.dll mozilla::detail::ConditionVariableImpl::wait mozglue/misc/ConditionVariable_windows.cpp:50
4 xul.dll mozilla::TaskController::GetRunnableForMTTask xpcom/threads/TaskController.cpp:586
5 xul.dll nsThread::ProcessNextEvent xpcom/threads/nsThread.cpp:1139
6 xul.dll NS_ProcessNextEvent xpcom/threads/nsThreadUtils.cpp:465
7 xul.dll nsThread::Shutdown xpcom/threads/nsThread.cpp:900
8 xul.dll nsThreadManager::ShutdownNonMainThreads xpcom/threads/nsThreadManager.cpp:391
9 xul.dll mozilla::ShutdownXPCOM xpcom/build/XPCOMInit.cpp:613

Comment 1

[Wayne Mery (:wsmwk)]

#8 crash for Firefox nightly

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The bug is linked to a topcrash signature, which matches the following criterion:

• Top 10 desktop browser crashes on nightly

For more information, please visit auto_nag documentation.

Comment 3

[Arthur K. (he/him)]

Seen and referenced in bug 1768344.

Comment 4

[Pascal Chevrel:pascalc]

Tracking given the volume on beta, Greg could we get this investigated please? Thanks

Comment 5

[Andrew McCreight [:mccr8]]

This signature is not great. I wonder if it used to show up as something else.

For beta crashes in the last week, 54% are hanging at XPCOMShutdownThreads, 20% at AppShutdownConfirmed, 11% at AppShutdownNetTeardown, 5% at AppShutdown.

Comment 6

[Andrew McCreight [:mccr8]]

I think this is just bug 1505660 with a new signature. (Well, also probably with some other signatures mixed in there.) The crashes there are completely gone in 106 beta.

Will, do you know if something happened that broke the shutdown hang signature improvement stuff (like bug 1402037)? Maybe just the inlining? Thanks.

Comment 7

[Andrew McCreight [:mccr8]]

(By "inlining", I mean how inlined functions can now show up in the signature.)

Comment 8

[Andrew McCreight [:mccr8]]

The crashes in bug 1682899 are also completely gone in 106 beta, so maybe the AppShutdownConfirmed hangs are that.

Comment 9

[Gabriele Svelto [:gsvelto]]

The first frame of interest that's not inlined in comment 0 is mozilla::TaskController::GetRunnableForMTTask(bool) and a few frames later mozilla::SpinEventLoopUntil(nsTSubstring<char> const&, nsThread::Shutdown::<lambda_7>&&, nsIThread*) so this would have probably fallen inside bug 1542485.

It seems that the signature changes are jumbling around these signatures, we probably might want to ignore or at least skip over mozilla::OffTheBooksCondVar::Wait() given it hides several different stacks above.

Comment 10

[Andrew McCreight [:mccr8]]

[Tracking Requested - why for this release]: This is more of an "untracking request". This looks like it is a jumble of old shutdown hang crashes being combined due to signature changes, so I don't think it needs to be tracked.

Comment 11

[Wayne Mery (:wsmwk)]

still seeing this on nightly updates
bp-95e2ff16-7076-45f0-87a8-2f0720220930
bp-93e2cbf3-4ef2-4d16-82fb-e3eba0220928

Perhaps related to mail accounts which are prompting for password but don't get logged in. I have removed them, and will see if that helps

Comment 12

[Jens Stutte [:jstutte]]

So aggregating by MOZ_CRASH_REASON over the last week yields:

Rank	Reason	#	%	Example	Bug/where
1	Shutdown hanging at step XPCOMShutdownThreads. Something is blocking the main-thread.	76	58.91 %	b8bf489a-6cbc-46c1-a6b3-b3d6d0221006	Bug 1505660
2	Shutdown hanging at step AppShutdownConfirmed. Something is blocking the main-thread.	12	9.30 %	7923fd06-56af-4118-9c97-271b70221006	nsJSInspector::EnterNestedEventLoop (called from JS: xpcInspector.enterNestedEventLoop(this);)
3	Shutdown hanging at step AppShutdownNetTeardown. Something is blocking the main-thread.	12	9.30 %	808c0987-71fc-4652-8bd0-f65270221005	Bug 1633342
4	Shutdown hanging at step AppShutdown. Something is blocking the main-thread.	10	7.75 %	7dc5302e-e5bc-4ddb-858d-b35010221006	CacheFileIOManager::Shutdown
5	Shutdown hanging at step XPCOMWillShutdown. Something is blocking the main-thread.	7	5.43 %	28d8203e-3053-4b5a-9d16-148af0221004	Bug 1710018
6	Shutdown hanging at step XPCOMShutdown. Something is blocking the main-thread.	2	1.55 %	5c37520d-6722-4834-ace5-8e8980220930	CanvasManagerParent::Shutdown
7	Shutdown hanging at step AppShutdownTeardown. Something is blocking the main-thread.	1	0.78 %	c9dfe588-bd1a-499f-96d0-f79ab0221006	TaskController::GetRunnableForMTTask does not get a next event

Comment 13

[Jens Stutte [:jstutte]]

Can we adjust the signatures accordingly?

Comment 14

[Jens Stutte [:jstutte]]

(In reply to Gabriele Svelto [:gsvelto] from comment #9)

The first frame of interest that's not inlined in comment 0 is mozilla::TaskController::GetRunnableForMTTask(bool) and a few frames later mozilla::SpinEventLoopUntil(nsTSubstring<char> const&, nsThread::Shutdown::<lambda_7>&&, nsIThread*) so this would have probably fallen inside bug 1542485.

No, the quota manager shutdown hangs have mozilla::SpinEventLoopUntil<mozilla::ProcessFailureBehavior::ReportToCaller, 'lambda at /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:2956:5'>(nsTSubstring<char> const&, mozilla::dom::quota::QuotaManager::Observer::Observe::<lambda_33>&&, nsIThread*) and keep arriving on bug 1542485, AFAICS. That signature falls into bug 1505660, as :mccr8 suspected in comment 6.

Comment 15

[Gabriele Svelto [:gsvelto]]

I filed bug 1794587 to adjust the signatures so they essentially go back to what they were before. In these crashes the addition of inlined functions to the signature provides not benefit, quite the opposite in fact.

Comment 16

[Gabriele Svelto [:gsvelto]]

I'm tempted to make the signature generation ignore mozilla::TaskController::GetRunnableForMTTask() as it doesn't seem to provide much value, it's just making our shutdown hang signatures longer. Jens what do you think about it? This would affect half a dozen bugs for which the signature would change.

Comment 17

[Jens Stutte [:jstutte]]

(In reply to Gabriele Svelto [:gsvelto] from comment #16)

I'm tempted to make the signature generation ignore mozilla::TaskController::GetRunnableForMTTask() as it doesn't seem to provide much value, it's just making our shutdown hang signatures longer. Jens what do you think about it? This would affect half a dozen bugs for which the signature would change.

Looks reasonable, it would be great if you can keep track of the needed adjustments to existing bug signatures.

Comment 18

[Gabriele Svelto [:gsvelto]]

(In reply to Jens Stutte [:jstutte] from comment #17)

Looks reasonable, it would be great if you can keep track of the needed adjustments to existing bug signatures.

I will! I can test the signatures beforehand thanks to socorro-siggen so I can add new signatures to the affected bugs even before we roll the changes into Socorro.

Comment 19

[Gabriele Svelto [:gsvelto]]

I've reviewed most of the affected crashes and there's almost nothing that needs to be done: in most cases the signatures will go back to what they were before we introduced inlined functions support.

Comment 20

[BugBot [:suhaib / :marco/ :calixte]]

The bug is linked to a topcrash signature, which matches the following criteria:

• Top 20 desktop browser crashes on release (startup)
• Top 20 desktop browser crashes on beta
• Top 10 desktop browser crashes on nightly
• Top 5 desktop browser crashes on Mac on beta
• Top 5 desktop browser crashes on Windows on beta

For more information, please visit auto_nag documentation.

Comment 21

[Jens Stutte [:jstutte]]

(In reply to Gabriele Svelto [:gsvelto] from comment #19)

I've reviewed most of the affected crashes and there's almost nothing that needs to be done: in most cases the signatures will go back to what they were before we introduced inlined functions support.

IIUC this change has yet to be done, right? The crashes seem to arrive still here. Thanks!

Comment 22

[Will Kahn-Greene [:willkg] ET needinfo? me]

I did a Socorro (crash ingestion pipeline and Crash Stats) prod deploy (bug #1797179) on 2022-10-24 which picked up signature changes. Looking at the graphs for both these signatures, there are no new reports since 2022-10-24, so I think we're done here.

Comment 23

[Jens Stutte [:jstutte]]

(In reply to Will Kahn-Greene [:willkg] ET needinfo? me from comment #22)

I did a Socorro (crash ingestion pipeline and Crash Stats) prod deploy (bug #1797179) on 2022-10-24 which picked up signature changes. Looking at the graphs for both these signatures, there are no new reports since 2022-10-24, so I think we're done here.

Oh, sorry for the noise, I'll just wait then to see the stats on the other bugs spike again.

Comment 24

[BugBot [:suhaib / :marco/ :calixte]]

Based on the topcrash criteria, the crash signatures linked to this bug are not in the topcrash signatures anymore.

For more information, please visit auto_nag documentation.

Comment 25

[Wayne Mery (:wsmwk)]

(In reply to Jens Stutte [:jstutte] from comment #23)

(In reply to Will Kahn-Greene [:willkg] ET needinfo? me from comment #22)

I did a Socorro (crash ingestion pipeline and Crash Stats) prod deploy (bug #1797179) on 2022-10-24 which picked up signature changes. Looking at the graphs for both these signatures, there are no new reports since 2022-10-24, so I think we're done here.

Oh, sorry for the noise, I'll just wait then to see the stats on the other bugs spike again.

I examined just a few bugs. The following increased since 2022-10-24

• Firefox Bug 1710018 - Crash in [@ shutdownhang | mozilla::PreferencesWriter::Flush]
• Thunderbird Bug 1524247 - Shutdown crash/hang due to endless wait loop when no password is entered | WaitForSingleObjectEx | WaitForSingleObject | _PR_MD_WAIT_CV | _PR_WaitCondVar | PR_WaitCondVar | mozilla::CondVar::Wait | nsEventQueue::GetEvent (but this is all version 91)

The following previously mentioned bugs did not increase

• Bug 1505660 - Crash in shutdownhang | nsThread::Shutdown | nsThreadManager::ShutdownNonMainThreads (but did increase greatly sept-october)
• Bug 1633342 - [meta] Crash in [mozilla::net::nsHttpConnectionMgr::Shutdown] and other net related places. Shutdown hang.

Comment 26

[BugBot [:suhaib / :marco/ :calixte]]

Based on the topcrash criteria, the crash signatures linked to this bug are not in the topcrash signatures anymore.

Since the crash volume is low (less than 5 per week), the severity is downgraded to S3. Feel free to change it back if you think the bug is still critical.

For more information, please visit auto_nag documentation.

Comment 27

[Jens Stutte [:jstutte]]

Signatures are back to normal since bug 1790983 landed, so let's close this.