1505660 - Crash in shutdownhang | nsThread::Shutdown | nsThreadManager::ShutdownNonMainThreads

Status: REOPENED

(Core :: XPCOM, defect, P2)

Description

[Trevor Skywalker]

This bug was filed from the Socorro interface and is
report bp-88218746-24bc-4e81-b9ce-d64cb0181106.
=============================================================

Top 10 frames of crashing thread:

0 ntdll.dll NtWaitForAlertByThreadId 
1 ntdll.dll RtlSleepConditionVariableSRW 
2 kernelbase.dll SleepConditionVariableSRW 
3 mozglue.dll mozilla::detail::ConditionVariableImpl::wait mozglue/misc/ConditionVariable_windows.cpp:58
4 xul.dll mozilla::ThreadEventQueue<mozilla::PrioritizedEventQueue<mozilla::EventQueue> >::GetEvent xpcom/threads/ThreadEventQueue.cpp:168
5 xul.dll nsThread::ProcessNextEvent xpcom/threads/nsThread.cpp:1171
6 xul.dll NS_ProcessNextEvent xpcom/threads/nsThreadUtils.cpp:530
7 xul.dll nsThread::Shutdown xpcom/threads/nsThread.cpp:953
8 xul.dll nsThreadManager::Shutdown xpcom/threads/nsThreadManager.cpp:341
9 xul.dll static nsresult mozilla::ShutdownXPCOM xpcom/build/XPCOMInit.cpp:926

=============================================================

Comment 1

[Andrew Overholt [:overholt]]

Jim, should we put shutdown hangs in a different component?

Comment 2

[Richard Leger]

FYI I observed two crashes today in TB version 65.0b2 (32-bit)

bp-dddcf79f-eab6-451c-a9f6-923c10190115 15/01/2019, 15:58
bp-9d9da253-2cbf-4896-8864-4542e0190115 15/01/2019, 12:42

I believe the first one occurred as I was trying to open a .eml attachment from a received email (opened in separate tab). IMAP/SMTP mailbox setup set to synchronise only 3 days worth of emails.

The second time when it happened I was not even actively using Thunderbird, but it was opened and close by itself... without my intervention :-)

Wayne I don't need info from you, it was just a way for me to let you know about crashes with my TB for your information as you seems to follow them up...

Comment 3

[Wayne Mery (:wsmwk)]

Your first crash better aligns with bug 1526127 - bp-dddcf79f-eab6-451c-a9f6-923c10190115 [@ shutdownhang | nsThread::Shutdown | nsThreadManager::Shutdown ] - trying to open a .eml attachment from a received email (opened in separate tab). IMAP/SMTP mailbox setup set to synchronise only 3 days worth of emails.

Your second crash is bug 1534119 - bp-9d9da253-2cbf-4896-8864-4542e0190115 [@ nsImapProtocol::HandleMessageDownLoadLine ] - not actively using Thunderbird

Comment 4

[Trevor Skywalker]

Top Crashers for Firefox 69.0a1 (Nightly) - 7 days ago

#13 0.45% 0.39% shutdownhang | nsThread::Shutdown | nsThreadManager::Shutdown 37 37 0 0 34 0 2017-09-27

Signature report for shutdownhang | nsThread::Shutdown | nsThreadManager::Shutdown

Showing results from 7 days ago

2,710 Results

Windows 10 1401 51.7%
Windows 7 1013 37.4%
Windows 8.1 187 6.9%
Windows Vista 81 3.0%
Windows 8 27 1.0%
Windows XP 1 0.0%

Product
Firefox 69.0a1 37 1.4% 46
Thunderbird 69.0a1 7 0.3% 5
Firefox 68.0b7 131 4.8% 113
Firefox 68.0b8 130 4.8% 133
Firefox 68.0b6 71 2.6% 81
Firefox 68.0b5 32 1.2% 34
Firefox 68.0b4 17 0.6% 20
Firefox 68.0b9 15 0.6% 13
Firefox 68.0b3 10 0.4% 9
Firefox 67.0.1 376 13.9% 234
Firefox 67.0 227 8.4% 179
Firefox 67.0.2 43 1.6% 42

Firefox 60.7.0esr 289 10.7% 285
Thunderbird 60.7.0 425 15.7% 313
Thunderbird 60.7.1 2 0.1% 2
Firefox 60.6.2esr 43 1.6% 43
Firefox 60.6.3esr 42 1.5% 37
Firefox 60.6.1esr 28 1.0% 21

Firefox 52.9.0esr 81 3.0% 41

Architecture
amd64 1465 54.1%
x86 1245 45.9%

Comment 5

[Wayne Mery (:wsmwk)]

Judging by release date of firefox 68, it looks like crash rate more than doubled.
https://crash-stats.mozilla.org/signature/?signature=shutdownhang%20%7C%20nsThread%3A%3AShutdown%20%7C%20nsThreadManager%3A%3AShutdown&date=>%3D2019-01-26T00%3A00%3A00.000Z&date=<2019-07-26T23%3A59%3A00.000Z#graphs

Comment 6

[Trevor Skywalker]

(In reply to Wayne Mery (:wsmwk) from comment #5)

Judging by release date of firefox 68, it looks like crash rate more than doubled.
https://crash-stats.mozilla.org/signature/?signature=shutdownhang%20%7C%20nsThread%3A%3AShutdown%20%7C%20nsThreadManager%3A%3AShutdown&date=>%3D2019-01-26T00%3A00%3A00.000Z&date=<2019-07-26T23%3A59%3A00.000Z#graphs

[@ shutdownhang | nsThread::Shutdown | nsThreadManager::Shutdown ]

There is still a high volume of crashes with this signature. It is the #2 Top Crasher on versions 68.0.1 and 68.0 (second only to the [@ OOM | small] signature). It is #3 Top Crasher on versions 69.0b13 and 60.7.2esr.

It is affecting the currently Nightly version of Firefox (70.0a1) and is the #19 Top Crasher for that version.

Could the priority for this bug be re-evaluated? I would suggest raising the priority from P2 to P1.

Could this bug be assigned to someone?

Thanks

Top Crashers for Firefox 68.0.1
Top 50 Crashing Signatures. 7 days ago

#1 9.01% -7.36% shutdownhang | nsThread::Shutdown | nsThreadManager::Shutdown 2225 2225 0 0 2101 0 2017-09-27

Top Crashers for Firefox 68.0.2
Top 50 Crashing Signatures. 7 days ago

#15 0.6% -0.02% shutdownhang | nsThread::Shutdown | nsThreadManager::Shutdown 645 645 0 0 647 0 2017-09-27

Top Crashers for Firefox 60.8.0esr
Top 50 Crashing Signatures. 7 days ago

#7 1.15% 0.02% shutdownhang | nsThread::Shutdown | nsThreadManager::Shutdown 318 318 0 0 274 0 2017-09-27

Top Crashers for Firefox 69.0b14
Top 50 Crashing Signatures. 7 days ago

#5 1.86% -1.59% shutdownhang | nsThread::Shutdown | nsThreadManager::Shutdown 193 193 0 0 180 0 2017-09-27

Top Crashers for Firefox 69.0b13
Top 50 Crashing Signatures. 7 days ago

#2 6.69% -5.61% shutdownhang | nsThread::Shutdown | nsThreadManager::Shutdown 177 177 0 0 149 0 2017-09-27

Top Crashers for Firefox 60.7.2esr
Top 50 Crashing Signatures. 7 days ago

#2 6.23% 1.25% shutdownhang | nsThread::Shutdown | nsThreadManager::Shutdown 168 168 0 0 155 0 2017-09-27

Top Crashers for Firefox 68.0
Top 50 Crashing Signatures. 7 days ago

#1 7.21% -0.95% shutdownhang | nsThread::Shutdown | nsThreadManager::Shutdown 148 148 0 0 139 0 2017-09-27

Top Crashers for Firefox 70.0a1
Top 50 Crashing Signatures. 7 days ago

#18 0.76% 0.63% shutdownhang | nsThread::Shutdown | nsThreadManager::Shutdown 55 55 0 0 54 0 2017-09-27

Signature report for shutdownhang | nsThread::Shutdown | nsThreadManager::Shutdown
Showing results from 7 days ago

Windows 10 2637 47.5%
Windows 7 1910 34.4%
Windows 8.1 886 15.9%
Windows Vista 62 1.1%
Windows 8 60 1.1%
Windows XP 2 0.0%

Firefox 68.0.1 2225 40.0% 1026
Firefox 68.0.2 646 11.6% 725
Firefox 60.8.0esr 318 5.7% 215
Firefox 69.0b14 192 3.5% 111
Firefox 69.0b13 182 3.3% 125
Firefox 60.7.2esr 168 3.0% 149
Firefox 68.0 147 2.6% 152
Firefox 69.0b15 82 1.5% 90
Firefox 60.6.2esr 74 1.3% 56
Firefox 52.9.0esr 65 1.2% 71
Firefox 69.0b12 65 1.2% 47
Firefox 70.0a1 55 1.0% 55
Firefox 69.0b11 51 0.9% 36

Uptime Range

1 hour 1643 29.6%
1-5 min 1507 27.1%
5-15 min 1287 23.2%
15-60 min 1117 20.1%
< 1 min 3 0.1%

Architecture
amd64 3924 70.6%
x86 1633 29.4%

Comment 7

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

The rate of shutdown hangs has recently doubled. Jimm, is the reason for this known?

Comment 8

[Wayne Mery (:wsmwk)]

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #7)

The rate of shutdown hangs has recently doubled. Jimm, is the reason for this known?

It is doubled mainly from increased crashes of Thunderbird 68.x, probably due to users updating from 60.x, being tracked in bug 1526127.
Per the graph https://crash-stats.mozilla.org/signature/?signature=shutdownhang%20%7C%20nsThread%3A%3AShutdown%20%7C%20nsThreadManager%3A%3AShutdown&date=%3E%3D2019-10-21T01%3A45%3A00.000Z&date=%3C2020-01-21T01%3A45%3A00.000Z#graphs

Comment 9

[Wayne Mery (:wsmwk)]

Thunderbird crashes should be referred to bug 1524247

Comment 10

[Jens Stutte [:jstutte]]

Looking at the aggregation of MOZ_CRASH_REASON I see here now:

1 	MOZ_CRASH(Shutdown too long, probably frozen, causing a crash.) 	24732 	100.00 %
2 	MOZ_CRASH(Shutdown hanging before starting.) 	1 	0.00 %

For the vast majority then, looking at the RunWatchdog this seems to indicate that:

1. All observers have been successfully notified and unblocked (sShutdownNotified is true)
2. No workers are hanging (runtimeService->CrashIfHanging(); did not crash)

To me this seems to indicate that there is nothing left worth waiting for (otherwise we would report it here) and we are just bloating our crash statistics. Still this is unexpected behavior, but for the end-user it should be handled without harm and I do not see how we could transform those crash reports into something more actionable if there is nothing concrete left we wait for.

A possible fix might be, to replace this MOZ_CRASH with a MOZ_ASSERT_UNREACHABLE and an exit(0) ? Or to dispatch a last "kill yourself now" message to the main thread and wait another second or two?

Comment 12

[Jens Stutte [:jstutte]]

The signature just added by the release bot seems to be a different situation. If I look at MOZ_CRASH_REASON there, I see only the MOZ_CRASH(Shutdown hanging before starting.) case, which is quite the opposite extreme (not even the first shutdown phase succeeded).

Sylvestre, can we prevent the bot somehow from adding this signature here?

Comment 13

[Sylvestre Ledru [:Sylvestre]]

Jens, the bot is doing that because bug 1678330 is marked as dup of this bug.

So, either you undup (and reopen) bug 1678330 or you remove the signature from bug 1678330

Comment 14

[Jens Stutte [:jstutte]]

Sorry, that was too easy...

Comment 15

[Jens Stutte [:jstutte]]

Attachment: Bug 1505660: Promote profile-change-net-teardown and -before-change-qm to have a shutdown timer reset. r?#dom-workers-and-storage,dthayer

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/45ba20e40892
Promote profile-change-net-teardown and -before-change-qm to have a shutdown timer reset. r=dthayer,mossop

Comment 17

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/45ba20e40892

Comment 18

[Jens Stutte [:jstutte]]

While the fix is expected to potentially reduce the number of crashes, it is not necessary a complete solution. We expect QM shutdown to be faster in general now thanks to bug 1666214, and we hope to have better diagnostics about the remaining cases through this patch here and will continue to observe this.

And for some of these crashes the MOZ_CRASH_REASON might just change slightly to be more explicit about the phase we are stuck in.

Comment 19

[BugBot [:suhaib / :marco]]

The patch landed in nightly and beta is affected.
:jstutte, is this bug important enough to require an uplift?
If not please set status_beta to wontfix.

For more information, please visit auto_nag documentation.

Comment 20

[Jens Stutte [:jstutte]]

Comment on attachment 9192576 [details]
Bug 1505660: Promote profile-change-net-teardown and -before-change-qm to have a shutdown timer reset. r?#dom-workers-and-storage,dthayer

Beta/Release Uplift Approval Request

• User impact if declined: Some users might experience less shutdown crashes thanks to this patch. And at least we might gain some more diagnostic insights on the different shutdown hangs in different phases.
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): The patch does only extend the explicit shutdown handling on two more shutdown phases. Normal operation (including successful shutdown) is unaffected.
• String changes made/needed:

Comment 21

[Julien Cristau [:jcristau] (back August 18)]

Comment on attachment 9192576 [details]
Bug 1505660: Promote profile-change-net-teardown and -before-change-qm to have a shutdown timer reset. r?#dom-workers-and-storage,dthayer

approved for 85.0b6

Comment 22

[Julien Cristau [:jcristau] (back August 18)]

Comment on attachment 9192576 [details]
Bug 1505660: Promote profile-change-net-teardown and -before-change-qm to have a shutdown timer reset. r?#dom-workers-and-storage,dthayer

https://hg.mozilla.org/releases/mozilla-beta/rev/f6ecda1b2d689320f4a3ce86893afb9efbbdef1a

[clearing approval to get this off the needs-uplift queries]

Comment 23

[Jens Stutte [:jstutte]]

FWIW, the current statistics show two interesting data points:

• Thunderbird is not appearing any more since mid-november. Is this just a change in reporting or did they actually solve something here?
• Since FF 94 each release causes a high spike, presumably right after the updates happened.

Comment 24

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Thunderbird crash reports: https://crash-stats.thunderbird.net/ is used for crash reports now, and https://crash-stats.mozilla.org/ rejects submissions for Thunderbird (bug 1608971).

Comment 25

[Wayne Mery (:wsmwk)]

(In reply to Jens Stutte [:jstutte] from comment #23)

FWIW, the current statistics show two interesting data points:

• Thunderbird is not appearing any more since mid-november. Is this just a change in reporting or did they actually solve something here?

Sadly, yes. crash-stats terminated supporting Thunderbird crashes. Collection stopped Nov 9.

There is public access to individual crash reports. But the detailed data on backtrace.io requires a paid account.

Comment 26

[Steven Michaud [:smichaud] (Retired)]

There is public access to individual crash reports. But the detailed data on backtrace.io requires a paid account.

How expensive is such an account? How do you get one? What's the public interface for searching this information, once you have an account?

I'm not necessarily going to go through with this. But the https://backtrace.io site has no information on this, or on Thunderbird at all. So it'd be nice to know.

Comment 27

[Wayne Mery (:wsmwk)]

responded to smichaud via PM

Comment 29

[BugBot [:suhaib / :marco]]

Copying crash signatures from duplicate bugs.

Comment 30

[David Parks [:handyman]]

crash-stats has the XPCOMSpinEventLoopStack field (under crash annotations) that is useful in this context. These signatures are going to be popular and will change over time but, as of today, I'm seeing:

@ shutdownhang | nsThread::Shutdown | nsThreadManager::Shutdown
By far the most popular cause is nsThread::Shutdown: BitsCommander, which is trying to stop a thread that is in the middle of a COM operation to do a background download. There are also BitsMonitor {some_guid} threads that are also related to downloads. I don't know if we expect to wait for or to kill incomplete downloads at shutdown. There are a few other stray thread names under this hang but most of the rest (about half) are just empty names. With those, it seems common to find BitsCommander doing COM on some thread (like with this crash ).

@ shutdownhang | mozilla::TaskController::GetRunnableForMTTask | nsThread::Shutdown | nsThreadManager::Shutdown
This only goes to Fx 84 -- all of the others are Thunderbird crashes. Most of the labels are empty -- maybe Tb doesn't label the relevant threads.

@ shutdownhang | nsThread::Shutdown | nsThreadManager::ShutdownNonMainThreads
Again, mostly BitsCommander threads not shutting down, although this signature brings in a healthy number of Update Watcher and TelemetryModule threads. There isn't a strong pattern here but the Telemetry hangs suggest AuthenticodeImpl::GetBinaryOrgName is too slow. See e.g. thread 24 of this crash. The Update Watcher thread is stuck waiting for the updater process to finish in nsUpdateProcessor::WaitForProcess.

Comment 31

[Jens Stutte [:jstutte]]

I scraped crashstats for nightly instances of this signature where some of the thread names contain SHDRCV and/or SHDACK (added by bug 1770451).

FWIW, here is the result.

There is definitely some correlation with the Updater process being waited on, as in bug 1772908. But we should also look at the other ones.

Comment 32

[Jens Stutte [:jstutte]]

There is also one interesting instance concerning the BHMgr Monitor and Cookie threads.

1. We enter phase ShutdownPhase::AppShutdown
2. (MainThread) notifies the CookieService to shutdown.
3. (MainThread) The CookieService starts a synchronous shutdown().
4. (MainThread) nsThread::Shutdown() is spinning the event loop, waiting for the thread to finish
5. (BHMgr Monitor) BackgroundHangManager::MonitorThread detects a hang and wants to report it (might have happened even earlier)
6. (BHMgr Monitor) BackgroundHangThread::ReportHang is called with mManager->mLock IS locked
7. (BHMgr Monitor) For whatever cosmic reason, writing to file takes some time inside nsLocalFile::OpenNSPRFileDesc
8. (MainThread) ProcessNextEvent wants to BackgroundHangMonitor().NotifyWait(); which wants to lock the same mManager->mLock already held by BackgroundHangThread::ReportHang and starves
9. (Cookie thread) In the meantime, the Cookie thread reached its end and is just waiting for the nsThreadShutdownAckEvent to be processed on the MainThread.

Also for performance reasons it seems questionable to me that a lock used by frequent event processing is held on the background hang monitor thread for long-running tasks like hang-report disk writings.

Bob, :jya, the lock that blocks the main thread has been introduced relatively recently (2 years ago) by bug 1634253.

Dragana, on a different side note the synchronous nsThread::Shutdown() in CookiePersistentStorage::Close could maybe be avoided in favor of nsThread::AsyncShutdown() ? It would not stop this from happening, of course, but there would be less blocking layers potentially stacking on the main thread.

Comment 33

[Olli Pettay [:smaug][[e-mail]]]

Doug, you you recall any of this stuff. Bug 1594577 changed some of the relevant code, though https://hg.mozilla.org/mozilla-central/annotate/2426bd765f8b74744fa7042826423ec5ddca53a7/xpcom/threads/nsThread.cpp#l1094 was added later.

Comment 34

[Bob Hood [:bhood]]

Andrew, I know this is a necro, but would you mind having a cursory look at this when you get a chance just in case Jean-Yves fails to respond? (See comment 32)

Comment 35

[Dragana Damjanovic [:dragana]]

I will open a bug to investigate a possible async shutdown of the cookie storage.

Comment 36

[Ray Kraesig [:rkraesig]]

A bit of additional archaeology and a bit of additional correlation.

---

(In reply to David Parks [:handyman] from comment #30)

crash-stats has the XPCOMSpinEventLoopStack field (under crash annotations) that is useful in this context. These signatures are going to be popular and will change over time

And indeed:

@ shutdownhang | nsThread::Shutdown | nsThreadManager::Shutdown

@ shutdownhang | nsThread::Shutdown | nsThreadManager::ShutdownNonMainThreads

These are effectively the same callstack, only before and after bug 1764119.

---

@ shutdownhang | mozilla::TaskController::GetRunnableForMTTask | nsThread::Shutdown | nsThreadManager::Shutdown
This only goes to Fx 84 -- all of the others are Thunderbird crashes. Most of the labels are empty -- maybe Tb doesn't label the relevant threads.

This is also the same callstack.

The primary difference — the presence of mozilla::TaskController::GetRunnableForMTTask — is spurious. The call stack is essentially identical in Fx and Tb, as can be seen in Visual Studio in the minidumps; but this frame is consistently inlined in Fx and consistently not inlined in Tb. Presumably this is due to some locally-uninteresting choice of compiler flags.

The secondary difference -- the absence of XPCOMSpinEventLoopStack annotations -- is mostly due to the fact that Tb didn't start writing these until about v95. As with Fx, the signature has changed to @ shutdownhang | mozilla::TaskController::GetRunnableForMTTask | nsThread::Shutdown | nsThreadManager::ShutdownNonMainThreads in recent versions, and XPCOMSpinEventLoopStack is generally present.

Thunderbird crashes with this signature are being investigated in bug 1768344 and bug 1749142.

(I've added the latter signature to the crash data, above, but perhaps I ought to have removed the former?)

---

More than 80% of these shutdown hangs appear to be in Thunderbird; of those, about 80% seem to be blocked on the "IMAP" thread. The remainder are stuck on "Update Watcher" and "TelemetryModule".

On the Firefox side (post-bug 1764119 only, for simplicity's sake), the vast majority of hangs are in "Update Watcher" and "BitsCommander", which are very nearly identical — it's not surprising that they're correlated, given that the only thing we use BITS for is downloading updates, but I have no explanation for why their ratio is approximately 1. The other hangs with this signature are mostly attributed to "TelemetryModule" and "Wifi Monitor", with "BitsMonitor {various uuids}" and "BHMgr Processor" being distant runners-up.

Comment 37

[Jens Stutte [:jstutte]]

(In reply to Jens Stutte [:jstutte] from comment #31)

I scraped crashstats for nightly instances of this signature where some of the thread names contain SHDRCV and/or SHDACK (added by bug 1770451).

FWIW, here is the result.

I updated the sheet.

Comment 38

[Olli Pettay [:smaug][[e-mail]]]

Moving to XPCOM, since many of the hangs are in the parent process.

Comment 39

[Jens Stutte [:jstutte]]

(In reply to Olli Pettay [:smaug][[e-mail]] from comment #38)

Moving to XPCOM, since many of the hangs are in the parent process.

Just to be precise: All of them.

Comment 40

[Jens Stutte [:jstutte]]

Moved ni?s to bug 1796060.

Comment 41

[Jens Stutte [:jstutte]]

(In reply to Jens Stutte [:jstutte] from comment #31)

I scraped crashstats for nightly instances of this signature where some of the thread names contain SHDRCV and/or SHDACK (added by bug 1770451).

FWIW, here is the result.

I updated the sheet. (Unfortunately this is not a simple query in crash-stats)

Comment 42

[Nick Alexander :nalexander [he/him] (PTO August 1-10)]

On the Firefox side (post-bug 1764119 only, for simplicity's sake), the vast majority of hangs are in "Update Watcher" and "BitsCommander", which are very nearly identical — it's not surprising that they're correlated, given that the only thing we use BITS for is downloading updates, but I have no explanation for why their ratio is approximately 1. The other hangs with this signature are mostly attributed to "TelemetryModule" and "Wifi Monitor", with "BitsMonitor {various uuids}" and "BHMgr Processor" being distant runners-up.

Jens observed that some of these have the BackgroundTaskName: backgroundupdate annotation. Such crashes during shutdown are exactly the kind of crash we would expect from background tasks: lots of Gecko doesn't have clean shutdown and background tasks can exit in a rush; there's often not much work to do. (And there's a timeout mechanism but last I checked, which was a while back, we don't see timeouts frequently.)

The two listed threads are very likely to be involved: backgroundupdate tasks install updates and interact with BITS (to start downloads and check on transfer progress).

Let me know if I can be of assistance debugging or reproducing these crashes. Thanks!

Comment 43

[Arthur K. (he/him)]

I hit this on 107.0b3 a bit ago. TB had presented me with the OAuth2-type username and PW dialogue but I was in the middle of shutting down TB when it hung and then crashed. So either an OAuth2 or PW manager issue.

https://crash-stats.mozilla.org/report/index/0a3b7415-dc4d-48d4-ba59-1eccf0221104

Comment 44

[Arthur K. (he/him)]

And a second shortly afterwards: https://crash-stats.mozilla.org/report/index/5dd1a9ea-54a5-4db2-8d65-1163e0221104

Maybe related to bug 1768344 since that's what I did before it crashed this time.

Comment 45

[Gabriele Svelto [:gsvelto]]

Added a signature

Comment 46

[Ray Kraesig [:rkraesig]]

@gsvelto: would it be feasible to bucket these crashes by their XPCOMSpinEventLoopStack value, rather than by their callstack? It should be both more stable and more indicative of the actual problem.

(Although at the moment it'd require some special-casing for BitsMonitor threads, due to the unfortunate embedded GUID.)

Comment 47

[Jens Stutte [:jstutte]]

(In reply to Nick Alexander :nalexander [he/him] from comment #42)

On the Firefox side (post-bug 1764119 only, for simplicity's sake), the vast majority of hangs are in "Update Watcher" and "BitsCommander", which are very nearly identical — it's not surprising that they're correlated, given that the only thing we use BITS for is downloading updates, but I have no explanation for why their ratio is approximately 1. The other hangs with this signature are mostly attributed to "TelemetryModule" and "Wifi Monitor", with "BitsMonitor {various uuids}" and "BHMgr Processor" being distant runners-up.

Jens observed that some of these have the BackgroundTaskName: backgroundupdate annotation. Such crashes during shutdown are exactly the kind of crash we would expect from background tasks: lots of Gecko doesn't have clean shutdown and background tasks can exit in a rush; there's often not much work to do. (And there's a timeout mechanism but last I checked, which was a while back, we don't see timeouts frequently.)

The two listed threads are very likely to be involved: backgroundupdate tasks install updates and interact with BITS (to start downloads and check on transfer progress).

Let me know if I can be of assistance debugging or reproducing these crashes. Thanks!

From recent nightly numbers (1 week) I see:

• 15 non-background task instances. All I looked at have to do with BitsCommander or UpdateWatcher.
• 172 background task ones, all of which have "backgroundupdate" as task name.

We should probably re-triage (and maybe rename) bug 1741675.

Comment 48

[Gabriele Svelto [:gsvelto]]

(In reply to Ray Kraesig [:rkraesig] from comment #46)

@gsvelto: would it be feasible to bucket these crashes by their XPCOMSpinEventLoopStack value, rather than by their callstack? It should be both more stable and more indicative of the actual problem.

Yes, it's possible and I'll file a bug for it.

(Although at the moment it'd require some special-casing for BitsMonitor threads, due to the unfortunate embedded GUID.)

That's a problem. For something to be in the signature it should be stable over time and crashes, otherwise bucketing won't work. We have steps within Socorro to sanitize stuff (mostly for privacy reason) but they add complexity and I don't think we want that in signature generation.

Comment 49

[Jens Stutte [:jstutte]]

(In reply to Jens Stutte [:jstutte] from comment #47)

From recent nightly numbers (1 week) I see:

• 15 non-background task instances. All I looked at have to do with BitsCommander or UpdateWatcher.
• 172 background task ones, all of which have "backgroundupdate" as task name.

We should probably re-triage (and maybe rename) bug 1741675.

This is still the case, it seems. And this is our second top-most crash in nightly, currently.

Comment 50

[Jens Stutte [:jstutte]]

Updating signatures to the active ones for current versions.

Comment 51

[Wayne Mery (:wsmwk)]

FWIW, most Thunderbird crashes are German locale (German 21%, English 19%, Japanese 15%, in that order).

Firefox crashes are overwhelmingly English - 38%

Comment 52

[Jens Stutte [:jstutte]]

Currently ~95% of the Firefox crashes are from backgroundupdate, see bug 1741675.

Comment 53

[Arthur K. (he/him)]

See https://bugzilla.mozilla.org/show_bug.cgi?id=1768344#c53

Comment 54

[Arthur K. (he/him)]

Since I am able to fairly easily able to repro the crash, is it worth it at all to get another perf profile as TB is shutting down after I do my STR? I can if you guys think it will help.

Comment 55

[Gian-Carlo Pascutto [:gcp]]

I'm going to unmark this main bug as S2, it makes more sense to prioritize the actual signatures.

Comment 56

[Jens Stutte [:jstutte]]

(In reply to Gian-Carlo Pascutto [:gcp] from comment #55)

I'm going to unmark this main bug as S2, it makes more sense to prioritize the actual signatures.

I agree, but we need some Sx value to make the triage bot happy.

Comment 57

[BugBot [:suhaib / :marco]]

The severity field for this bug is set to S3. However, the following bug duplicate has higher severity:

• Bug 1769211: S2

:jstutte, could you consider increasing the severity of this bug to S2?

For more information, please visit BugBot documentation.

Comment 58

[Jens Stutte [:jstutte]]

No, see comment 55.

Comment 59

[BugBot [:suhaib / :marco]]

The leave-open keyword is there and there is no activity for 6 months.
:jstutte, maybe it's time to close this bug?
For more information, please visit BugBot documentation.

Comment 60

[BugBot [:suhaib / :marco]]

Based on the topcrash criteria, the crash signatures linked to this bug are not in the topcrash signatures anymore.

For more information, please visit BugBot documentation.

Comment 61

[Jens Stutte [:jstutte]]

Looking at the top-ten nsThreadShutdown SpinEventLoopUntil reasons:

Rank	Thread	#	%	Bug
1	default: nsThread::Shutdown: BitsCommander - most of these in the backgroundupdate process	4085	50.58 %	Bug 1741675
2	default: RemoteWorkerService::Observe nsThread::Shutdown: ProxyResolution	1190	14.74 %	Bug 1976693
3	default: nsThread::Shutdown: Socket Thread	550	6.81 %	Bug 1863599 (?!?)
4	default: nsThread::Shutdown: UpdateProcessor	545	6.75 %	Only ESR 115
5	default: nsThread::Shutdown: Wifi Monitor	119	1.47 %	Bug 1976696
6	default: nsThread::Shutdown: ImageIO	111	1.37 %
7	default: AsyncShutdown Spinner for profile-before-change nsThread::Shutdown: sqldb:places.sqlite #1	88	1.09 %
8	default: AsyncShutdown Spinner for profile-before-change nsThread::Shutdown: sqldb:places.sqlite #2	81	1.00 %
9	default: AsyncShutdown Spinner for profile-before-change nsThread::Shutdown: sqldb:places.sqlite #3	79	0.98 %
10	default: nsThreadPool::ShutdownWithTimeout nsThread::Shutdown: ProxyResolution	77	0.95 %	Bug 1976693

(note that the query catches all nsThreadShutdown hangs regardless of the phase).

Comment 62

[Jens Stutte [:jstutte]]

Top ten of the last month:

1 	default: nsThread::Shutdown: BitsCommander 	1810 	42.33 %
2 	default: RemoteWorkerService::Observe|nsThread::Shutdown: ProxyResolution 	684 	16.00 %
3 	default: nsThread::Shutdown: Socket Thread 	295 	6.90 %
4 	default: nsThread::Shutdown: ImageIO 	168 	3.93 %
5 	default: nsThread::Shutdown: Wifi Monitor 	111 	2.60 %
6 	default: AsyncShutdown Spinner for profile-before-change|nsThread::Shutdown: sqldb:places.sqlite #3 	80 	1.87 %
7 	default: AsyncShutdown Spinner for profile-before-change 	76 	1.78 %
8 	default: AsyncShutdown Spinner for profile-before-change|nsThread::Shutdown: sqldb:places.sqlite #2 	70 	1.64 %
9 	default: RemoteWorkerService::Observe|nsThread::Shutdown: DOM Worker|nsThread::Shutdown: ProxyResolution 	60 	1.40 %
10 	default: AsyncShutdown Spinner for profile-change-teardown 	57 	1.33 %

Note that the top 3 are still the same as in comment 61.

Comment 63

[Jens Stutte [:jstutte]]

Keeping this blocking old closed bugs has no meaning.

Comment 64

[Worcester12345]

(In reply to Jens Stutte [:jstutte] from comment #63)

Keeping this blocking old closed bugs has no meaning.

... unless they get reopened, then the meaning is lost.

Comment 65

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

There is a spike in crashes with [@ shutdownhang | mozilla::SpinEventLoopUntil | nsThread::WaitForAllAsynchronousShutdowns | nsThreadManager::ShutdownNonMainThreads] (crash-stats) for Firefox 149.0.2 (spike for v127 excluded) for the last two days. Could it be affected by the upgrade to Firefox 150?

Comment 66

[Wayne Mery (:wsmwk)]

We've also been tracking the same spike in Thunderbird. Not sure of the origin because we haven't dug in too deeply. But my guess is something external is involved, because 149.0.2 has been out for a while.

I haven't seen any Thunderbird user comments mentioning updates. But in Firefox crashes there are many comments about updating.

Comment 67

[Jens Stutte [:jstutte]]

(In reply to Aryx (:aryx) from comment #65)

There is a spike in crashes with [@ shutdownhang | mozilla::SpinEventLoopUntil | nsThread::WaitForAllAsynchronousShutdowns | nsThreadManager::ShutdownNonMainThreads] […] for Firefox 149.0.2 […] Could it be affected by the upgrade to Firefox 150?

Confirmed, and yes — triggered by the 150 rollout, but the magnitude may be caused by a concurrent Windows 11 regression rather than anything on our side, at least I could not find relevant code changes. Summary of what I found:

This signature is a catch-all. It collapses every parent-process hang where some nsThread failed to ack its AsyncShutdown by the time we reach xpcom-shutdown-threads. nsThreadManager::ShutdownNonMainThreads fires AsyncShutdown on every remaining nsThread in a single batch (nsThreadManager.cpp#425) and then main-thread spins on WaitForAllAsynchronousShutdowns (nsThreadManager.cpp#433), whose reason string (nsThread.cpp#903) is just "nsThread::WaitForAllAsynchronousShutdowns" — no thread name. So xpcom_spin_event_loop_stack on these crashes is uniformly default: nsThread::WaitForAllAsynchronousShutdowns (99.7% of the spike crashes), and the signature tells us nothing about which thread is the straggler.

For the Apr 21-22 spike, the straggler is BitsCommander in 100% of sampled crashes. 30/30 spike crashes on 149.0.2 and 15/15 on 149.0 have a live BitsCommander thread; 0/30 matched-version non-shutdownhang parent crashes do. The stack is identical across every sample — a synchronous COM RPC to the Windows BITS service that never returns:

NtAlpcSendWaitReceivePort
LRPC_BASE_CCALL::DoSendReceive
LRPC_CCALL::SendReceive
I_RpcSendReceive
CMessageCall::RpcSendRequestReceiveResponse
CSyncClientCall::SendReceive{2,InRetryContext,…}
NdrExtpProxySendReceive
NdrpClientCall3

BitsService::shutdown_command_thread calls AsyncShutdown when the last in-flight request drops off (bits_interface/mod.rs#110). That bumps mOutstandingShutdownContexts on the calling thread. When BitsCommander is stuck in the RPC above it never processes the shutdown event, the ack never comes back, and WaitForAllAsynchronousShutdowns hangs until the watchdog fires.

Same root cause as bug 1741675, but a different signature than listed there — 1741675 is the default: nsThread::Shutdown: BitsCommander variant which comes from the synchronous nsIThread::Shutdown() code path (mostly backgroundupdate process). The UI process uses the batch async path, and the RPC hang lands here instead.

The rollout is the trigger, but not the explanation. 150.0 shipping caused UI 149.0.2 instances to stage the MAR via BITS, so BitsCommander became a common live thread during shutdown in the UI process at scale. But raw hang counts on this signature across the last three update cycles show the 149.0.2 → 150 cycle is ~6× what prior cycles produced:

| Cycle                 | Day    | Version | Hangs           |
|-----------------------|--------|---------|-----------------|
| 148 → 149 (major)     | Mar 25 | 148.0.2 | 59              |
|                       | Mar 26 |         | 69              |
|                       | Mar 27 |         | 100             |
|                       | Mar 28 |         | 56              |
| 149 → 149.0.2 (dot)   | Apr 8  | 149.0   | 140             |
|                       | Apr 9  |         | 139             |
|                       | Apr 10 |         | 80              |
| 149.0.2 → 150 (major) | Apr 21 | 149.0.2 | 1292            |
|                       | Apr 22 |         | 2002            |
|                       | Apr 23 |         | 538 (partial)   |

The prior major (148 → 149) looks like the dot release — so this is not a "major update brings a bigger MAR" story. We also checked: no commits in toolkit/components/bitsdownload/ or toolkit/mozapps/update/ between 149.0 and 149.0.2.

What does explain the 6× inflation is a Windows-version skew. Breakdown of the spike signature vs. all 149.0.2 parent crashes on Windows (Apr 21-22):

| Windows build | Hang crashes | All parent | Share of parent |
|---------------|--------------|------------|-----------------|
| 10.0.26200    | 3572 (93.2%) | 7996       | 44.67%          |
| 10.0.26100    | 179  (4.7%)  | 1019       | 17.57%          |
| 10.0.22631    | 3            | 607        | 0.49%           |
| 10.0.19045    | 52   (1.4%)  | 4721       | 1.10%           |

A user on Windows 11 build 26200 is ~40× more likely to hit this hang than a Windows 10 user on the same Firefox, and ~90× more likely than a Windows 11 23H2 user. Build 26100 (the other Windows 11 24H2 family build) is also elevated. This strongly suggests a BITS service regression in a recent Windows 11 cumulative update landing in the same window, making CSyncClientCall::SendReceive block far longer than before. The 150 rollout is the trigger that brought enough simultaneous in-flight BITS jobs into existence for us to notice.

Comment 68

[Jens Stutte [:jstutte]]

To be clear, these hangs all happen in the backgroundupdate background task and I assume/hope it will just work next time, but the volume is really high.

Actually the aggregate in socorro's UI hides values for a field if they have none. So 90% seem to happen on the real parent process, IIUC.

Comment 69

[Wayne Mery (:wsmwk)]

Thunderbird is peaking at 4,000 to 5,000 crashes per day for https://crash-stats.mozilla.org/signature/?signature=shutdownhang%20%7C%20mozilla%3A%3ASpinEventLoopUntil%20%7C%20nsThread%3A%3AWaitForAllAsynchronousShutdowns%20%7C%20nsThreadManager%3A%3AShutdownNonMainThreads&date=%3E%3D2026-03-28T00%3A00%3A00.000Z&date=%3C2026-04-28T23%3A59%3A00.000Z#graphs

Comment 70

[Corey Bryant]

(In reply to Jens Stutte [:jstutte] from comment #67)
...

A user on Windows 11 build 26200 is ~40× more likely to hit this hang than a Windows 10 user on the same Firefox, and ~90× more likely than a Windows 11 23H2 user. Build 26100 (the other Windows 11 24H2 family build) is also elevated. This strongly suggests a BITS service regression in a recent Windows 11 cumulative update landing in the same window, making CSyncClientCall::SendReceive block far longer than before. The 150 rollout is the trigger that brought enough simultaneous in-flight BITS jobs into existence for us to notice.

Jens, do you know if a bug has been opened for Microsoft to take a look?

Comment 71

[Wayne Mery (:wsmwk)]

(In reply to Wayne Mery (:wsmwk) from comment #69)

Thunderbird is peaking at 4,000 to 5,000 crashes per day

Thunderbird crash rate tripled starting 4/28, the day after update rate of 150.0 was changed to 100. TBD whether it will crash rate taper off this week when users successfully(??) update.

Comment 72

[Wayne Mery (:wsmwk)]

Correction ...

(In reply to Wayne Mery (:wsmwk) from comment #71)

(In reply to Wayne Mery (:wsmwk) from comment #69)

Thunderbird is peaking at 4,000 to 5,000 crashes per day

Thunderbird release channel crash rate doubled to ~6k per day starting 4/28, the day after update rate of 150.0 was changed to 100, with 668 user comments. TBD whether it will crash rate taper off this week when users successfully(??) update.

Comment 73

[Corey Bryant]

Hi Jens, Is there a bug opened for Microsoft to take a look at this?

Comment 74

[Jens Stutte [:jstutte]]

(In reply to Corey Bryant from comment #73)

Hi Jens, Is there a bug opened for Microsoft to take a look at this?

I am not aware of anything. Chris ?

Comment 75

[Wayne Mery (:wsmwk)]

Attachment: Screenshot 2026-05-24 at 4.13.47 PM.png

As of ~May 14, the crash rate dropped back to previous levels.

Comment 76

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Since 153.0a1 20260523100109, [@ shutdownhang | RtlWaitOnAddress | WaitOnAddress] has some crash reports for each build. The stack contains mozilla::SpinEventLoopUntil(nsTSubstring<char> const&, nsThread::WaitForAllAsynchronousShutdowns::<lambda_2>&&, nsIThread*).

Comment 77

[Pascal Chevrel (relman team) -> :pascalc|PTO until Sept 2]

Jens, this is spiking again with 153.0

Comment 78

[Jens Stutte [:jstutte]]

Apparently only on Windows. I am traveling today, Yannis, can you take alook?

Comment 79

[Yannis Juglaret [:yannis]]

The spike is a volume transfer from old signature shutdownhang | mozilla::SpinEventLoopUntil | nsThreadPool::ShutdownWithTimeout (tracked by bug 1866944). In that bug you'll see we had huge volume in 152 and none in 153. The volume transfer occurs because of the low-level implementation changes that landed in bug 2015967.

Like bug 1866944 comment 30 already figured, the bulk of the spike is the taskbar spin issue fixed by bug 1977202 in 154 branch. So a patch is ready if we want an uplift. I'll ask directly there if the patch is uplift-ready.

Comment 80

[Wayne Mery (:wsmwk)]

(In reply to Yannis Juglaret [:yannis] from comment #79)

The spike is a volume transfer from old signature shutdownhang | mozilla::SpinEventLoopUntil | nsThreadPool::ShutdownWithTimeout (tracked by bug 1866944). In that bug you'll see we had huge volume in 152 and none in 153. The volume transfer occurs because of the low-level implementation changes that landed in bug 2015967.

Like bug 1866944 comment 30 already figured, the bulk of the spike is the taskbar spin issue fixed by bug 1977202 in 154 branch. So a patch is ready if we want an uplift. I'll ask directly there if the patch is uplift-ready.

In this case, it became a topcrash for Thunderbird

Comment 81

[Wayne Mery (:wsmwk)]

Attachment: Screenshot 2026-08-06 at 9.47.51 AM.png

Comment 82

[Wayne Mery (:wsmwk)]

Attachment: v153b and v153b crash rate

As measured by v153b and v153b crash rate, it's hard to see that the v154 patch in bug 1977202 has made any difference.

Comment 83

[Jens Stutte [:jstutte]]

Leaving this AI analysis here, mostly to explain the changes in signatures. If we really want to do the signature split is probably TBD.

Some signature bookkeeping first, because I think comments #79-#82 are partly
talking about two different hangs.

The signature this bug tracks is at zero. shutdownhang | nsThread::Shutdown | nsThreadManager::ShutdownNonMainThreads has 0 crashes in both Firefox and
Thunderbird over 2026-06-16 .. 2026-08-06. Everything moved to
shutdownhang | RtlWaitOnAddress | WaitOnAddress after bug 2015967 (landed
2026-05-23) put the low-level wait on the Windows futex path. So this bug carries
top50 + topcrash-thunderbird against a signature that no longer fires, while the
volume being discussed lives under a signature that isn't listed here. That alone
makes the numbers hard to compare across comments.

What the new umbrella actually contains. Splitting it by proto_signature,
2026-07-01 .. 2026-08-06:

main-thread wait	Firefox	Thunderbird
nsThreadPool::ShutdownWithTimeout / BackgroundEventTarget::Shutdown	8,307 (60%)	410 (5%)
nsThread::WaitForAllAsynchronousShutdowns	246 (1.8%)	6,540 (82%)
(totals)	13,796	7,987

The two products are almost perfectly inverted. Bug 1977202
(PinCurrentAppToTaskbarWin11 blocking thread pool shutdown) targets the pool
path, i.e. Firefox's 60%, and it is Firefox::Shell Integration code that
Thunderbird does not run. Thunderbird's 82% is the
WaitForAllAsynchronousShutdowns path, which is this bug's original subject.

Two notes on the graph in comment #82: it is filtered to 155.0a1 / 154.0b /
153.0b / 154.0a1, which is roughly 6% of the signature's volume (~1,300 of
~22,300 crashes for 2026-06-16 .. 2026-08-06); the Thunderbird topcrash itself
sits in 153.0.1 / 153.0 / 153.0.1esr, none of which are in that filter. And
1977202 is firefox153: wontfix, with the esr153 uplift only landing 2026-07-26,
while 98% of the pool-path volume is on 153.0 / 153.0.1 / 153.0.3 release. So
neither the release population (no patch) nor the nightly/beta lines (mixed fixed
and unfixed versions, aggregated into one line per product) can show whether that
fix worked. I am not claiming it did or did not help - just that this view cannot
answer it.

Proposal: split the signature. The signature lists already contain everything
needed; the walk just stops one frame too early. Frames are matched with
re.compile("|".join(lines)) plus .match()
(socorro/signature/rules.py), so entries are leading-prefix matches:

ZwWaitForAlertByThreadId -> irrelevant, skipped
RtlWaitOnAddress -> matches the bare Rtl prefix entry, kept and continue
WaitOnAddress (kernelbase, no Rtl) -> in neither list, terminates here
below it, mozilla::FutexImpl<T>::wait, then ConditionVariableImpl::wait*,
OffTheBooksCondVar::Wait, TaskController::GetRunnableForMTTask,
nsThread::ProcessNextEvent, NS_ProcessNextEvent - all already irrelevant,
and mozilla::SpinEventLoopUntil, nsThread::Shutdown, .*WaitFor all
already prefixes. None of it is ever consulted.

Because Rtl is a prefix rather than irrelevant, the same hang also fragments by
how deep the OS-internal wrappers go. Last 7 days:

shutdownhang | RtlWaitOnAddress | WaitOnAddress - 7,738 Firefox
shutdownhang | RtlpWaitOnAddressWithTimeout | RtlpWaitOnAddress | RtlWaitOnAddress | WaitOnAddress - 5,998 Firefox, 2,733 Thunderbird

One hang, split roughly 57/43, so its actual ranking is understated by nearly
half.

Suggested addition to irrelevant_signature_re.txt (irrelevant is tested before
prefix, so these win over Rtl):

Rtlp?WaitOnAddress(WithTimeout)?
WaitOnAddress
mozilla::FutexImpl

Expected result for the two dominant paths:

Firefox -> shutdownhang | mozilla::SpinEventLoopUntil | nsThreadPool::ShutdownWithTimeout
Thunderbird -> shutdownhang | mozilla::SpinEventLoopUntil | nsThread::WaitForAllAsynchronousShutdowns | nsThreadManager::ShutdownNonMainThreads

That is the pre-2015967 shape, so it both separates the two products
automatically and restores continuity with the long history under the old
signatures instead of leaving us with a volume-transfer argument.

Worth including at the same time, both visible in the current tail:
(anonymous namespace)::LockContended (otherwise the
TimerThread::HiResWindowsMonitor stacks stop one frame later), and the Rust std
futex chain std::sys::pal::windows::futex::(wait_on_address|futex_wait),
std::sys::sync::mutex::futex::Mutex::lock, std::sync::poison::mutex::Mutex,
which currently hides a glean with_glean lock under KillClearOnShutdown.

Caveats: these lists are global, so WaitOnAddress becoming irrelevant affects
any crash whose crashing thread sits on a futex - that is the same treatment
WaitForSingleObject and RtlSleepConditionVariableSRW already get, but it is
not scoped to shutdown hangs. Existing crashes are not re-signatured without a
reprocessing request. And the ceiling on all of this: the main-thread stack tells
us which wait we are stuck in, never which thread failed to shut down, so
Thunderbird's 82% still needs the non-crashing threads from full minidumps.

If there are no objections I will file this against Socorro :: Signature and add
the new signature(s) to this bug.