1257058 - Shutdown crash/hang due to endless wait loop when no password is entered | WaitForSingleObjectEx | WaitForSingleObject | _PR_MD_WAIT_CV | _PR_WaitCondVar | PR_WaitCondVar | mozilla::CondVar::Wait | nsEventQueue::GetEvent

Status: VERIFIED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[Wayne Mery (:wsmwk)]

Attachment: TB45 pwd 2 crash windbg analyze -f

Fully reproducible testcase for bug 1149287. 
Tested on win7 laptop, wifi.

1. two accounts defined one imap, one pop
2. no saved passwords, no master password
3. start thunderbird
4. when prompted for the imap password, don't type anything and click OK
5. when prompted for the pop password, click cancel
6. shutdown

TB 31 hangs.  (note: NO hang if pop pwd prompt is given OK and imap is cancelled)

TB 38 and newer crashes after the shutdown hang timer expires. 
Windbg stack attached for TB45 "hang", presumable before the shutdown timer expired. 

bp-f96eda58-b469-458a-b36e-4f1e92160315 TB45 crashes
 0 	xul.dll	mozilla::`anonymous namespace'::RunWatchdog(void*)	toolkit/components/terminator/nsTerminator.cpp
1 	nss3.dll	_PR_NativeRunThread	nsprpub/pr/src/threads/combined/pruthr.c
2 	nss3.dll	pr_root	nsprpub/pr/src/md/windows/w95thred.c
3 	msvcr120.dll	_callthreadstartex	f:\dd\vctools\crt\crtw32\startup\threadex.c:376
4 	msvcr120.dll	msvcr120.dll@0x2c000	
5 	kernel32.dll	BaseThreadInitThunk	
6 	ntdll.dll	__RtlUserThreadStart	
7 	ntdll.dll	_RtlUserThreadStart

Comment 1

[Wayne Mery (:wsmwk)]

clarifying with additional variations:
- no failure if a password is typed and OK is clicked, then shutdown
- no failure of incorrect password is typed+OK, then wait for "Login to server xxx failed.", then cancel or retry with success
- fails when incorrect pwd+OK, then "retry", blank out pwd, click OK, shutdown

I did not any combination of multiple accounts all waiting for password, with or with master password, which would cause Thunderbird to hang or crash after long periods of unanswered prompts. (i.e. PRIOR to manual shutdown)

Comment 2

[Wayne Mery (:wsmwk)]

p.s. clean profile

same results for TB24.6.0 and TB12.0.1

Comment 3

[Kent James (:rkent)]

Let's try to investigate this in a point release of TB 45.

Comment 4

[Wayne Mery (:wsmwk)]

Also gotten this previously when not at my desk. bp-cca66132-7920-4c6f-bcfd-e355e2160302

Comment 5

[Wayne Mery (:wsmwk)]

Chiaki/Matt can you reproduce with this steps of comment 0?

Comment 6

[ISHIKAWA, Chiaki]

(In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment #5)
> Chiaki/Matt can you reproduce with this steps of comment 0?

Wayne, 
I tested tb 45 linux 64-bit.
Crashed exactly as you describe.
Signature.
bp-fa19d074-9167-4138-b804-84aaf2160422	2016年04月22日
bp-07905e99-18bc-4cab-8ded-4268e2160422	2016年04月22日
bp-5466b37d-6595-46ea-bd4d-732002160422	2016年04月22日

Actually, in one of the crashes, I made a mistake of swapping OK/click to the prompts (I think the first one, which is the one at the bottom?), but TB still crashed in the same manner: after hitting the quit button and then wait for 60 seconds for timeout and
then I see the crash reporter.

Hope this helps.

TIA

Comment 7

[Wayne Mery (:wsmwk)]

Experimented with several releases. 7.1.0, 9.0.1 and others hang with progress bar spinning for about a minute after file|exit or X. 
* After ~60 seconds if I shut Thunderbird immediately after clicking on password prompt.  
* After 30 seconds if I let Thunderbird sit a spell before shutting Thunderbird. 

So if toolkit.asyncshutdown.crash_timeout were 61 seconds some crashes would be prevented because the hang would be cleared. Not not a good first choice. Why the hang? 

Dug deep looking for a possible regression. 3.1.18 does not hang. 
TB 3.3a2pre 2011-01-05 hangs 60 sec
TB 3.3a2pre 2011-01-04 no hang (task gone within 5 sec)
regression range http://hg.mozilla.org/comm-central/pushloghtml?startdate=2011-01-04%2003:05:00&enddate=2011-01-05%2003:05:00

There may be more than one factor involved. Is it imap, or pop, or something else.
But perhaps we can suspect one part being bug 613184!  
Didn't dig if related to one of my hang reports http://mzl.la/1rqoJQl

Comment 8

[Wayne Mery (:wsmwk)]

Attachment: TB48 hang windbg analyze -hang

captured TB state before timeout caused crash (to supplement attachment 8731026 [details])

Comment 9

[Wayne Mery (:wsmwk)]

(In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment #8)
> Created attachment 8744415 [details]
> TB48 hang windbg analyze -hang
stacktrace was done with 2016-04-04 nightly, which does not have fix for bug 698882, which landed 2016-03-28
 Deadlock in nsSocketTransportService [tbird] PR_SetPollableEvent

Tested with 2016-04-22 nightly.  crashes immediately on shutdown bp-ad78cecb-7c53-4cba-9496-ff6082160422
So bug 698882 didn't helpe me.

Comment 10

[Wayne Mery (:wsmwk)]

Magnus, what options do we have here, or what do you see as the next step?

Comment 11

[Wayne Mery (:wsmwk)]

somewhere in version 45, this signature becomes #1, far overtaking bug 1149287

looking at crashes with email addresses in past 5 months, 90% of users crash only once ... and only with shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | _PR_MD_WAIT_CV | _PR_WaitCondVar | PR_WaitCondVar | mozilla::CondVar::Wait | nsEventQueue::GetEvent

but the few who have multiple crashes, like cem19, also see #27 crash 
shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | _PR_MD_WAIT_CV | _PR_WaitCondVar | PR_WaitCondVar | PR_Wait | PR_CWait | nsMsgCopy::DoCopy

Comment 12

[Wayne Mery (:wsmwk)]

At least one user's crashes morphed to  shutdownhang | _PR_MD_WAIT_CV | _PR_WaitCondVar | mozilla::CondVar::Wait | nsEventQueue::GetEvent | nsThread::ProcessNextEvent | NS_ProcessNextEvent | nsThread::Shutdown | nsThreadManager::Shutdown 
bp-da5fb252-dfb0-43d9-8e77-fbc192160923

Comment 14

[Makoto Kato [:m_kato]]

ReentrantMonitorAutoEnter into nsImapProtocol::GetPassword should finish loop when shutdown observer occur.

Comment 15

[Wayne Mery (:wsmwk)]

Thanks m_kato!

Now that we have a solid diagnosis, we just need someone to code it - to hopefully kill our 3 of our top 5 crashers. And hopefully a couple more lower ranked signatures

The WaitForSingleObjectEx  signature has changed to
win10 shutdownhang | ntdll.dll@0x6dd3c  #1 crash
win7  shutdownhang | ntdll.dll@0x1f8e1  #3 crash
win7  shutdownhang | ntdll.dll@0x46bf4  #6 crash 
win10 shutdownhang | ntdll.dll@0x93930  #11 crash

I would be very surprised if this didn't also affect seamonkey - but I don't find these crash signature for them.

Comment 16

[Rainer Bielefeld]

Blocking bug 1316927 and bug 1149287 this one can't be NT only?

Comment 17

[Wayne Mery (:wsmwk)]

The win10 signature is now shutdownhang | ntdll.dll@0x3c33c

Comment 19

[Wayne Mery (:wsmwk)]

version 60 signatures
shutdownhang | arena_t::GetNonFullBinRun | nsThread::Shutdown | nsThreadManager::Shutdown
shutdownhang | mozilla::Queue<T>::Push
shutdownhang | nsCOMPtr_base::assign_with_AddRef | mozilla::detail::MutexImpl::unlock | nsThread::Shutdown | nsThreadManager::Shutdown  bp-4c05f616-c4c8-4d3a-a3f0-aeadc0180810

Comment 20

[Wayne Mery (:wsmwk)]

I need to retest this with version 60

Comment 21

[Wayne Mery (:wsmwk)]

(In reply to Makoto Kato [:m_kato] from comment #14)
> ReentrantMonitorAutoEnter into nsImapProtocol::GetPassword should finish
> loop when shutdown observer occur.

#1 crash for 60.2.1 - 15% of all crashes, now that master password bug 1176399 is fixed in 60.2.1.

I assume this is still highly reproducible per comment 0

Comment 22

[Jorg K (CEST = GMT+2)]

https://crash-stats.mozilla.com/signature/?signature=shutdownhang%20%7C%20nsCOMPtr_base%3A%3Aassign_with_AddRef%20%7C%20mozilla%3A%3Adetail%3A%3AMutexImpl%3A%3Aunlock%20%7C%20nsThread%3A%3AShutdown%20%7C%20nsThreadManager%3A%3AShutdown&date=%3E%3D2018-09-30T20%3A59%3A08.000Z&date=%3C2018-10-07T20%3A59%3A08.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=install_time&_sort=-date&page=1#reports

Dies deep down in Mozilla code :-(

Comment 23

[Jorg K (CEST = GMT+2)]

I'll take a look at comment #0 and comment #14. Perhaps if it's just about cancelling a password loop, it could be an "easy" fix.

Comment 24

[Jorg K (CEST = GMT+2)]

Original STR:
1. two accounts defined one imap, one pop
2. no saved passwords, no master password
3. start thunderbird
4. when prompted for the imap password, don't type anything and click OK
5. when prompted for the pop password, click cancel
6. shutdown

I did this simplified version:
1. one IMAP account defined
2. no saved passwords, no master password
3. start thunderbird
4. when prompted for the imap password, don't type anything and click OK
5. shutdown

I tried to start TB again on the same profile, but couldn't can't since it was still hanging in there. After a while, the crash reporter came up and I reported bp-c74f4e7f-78bf-4474-a59e-15d600181008.

Fully reproducible. Patch should be easy, I'll look into it in the next 24 hours unless someone beats me to it.

Comment 25

[Jorg K (CEST = GMT+2)]

Attachment: 1257058-shutdown-crash.patch

This implements a shutdown listener and the wait look will exist. Sadly it completes the current wait of one minute (kImapSleepTime = PR_MillisecondsToInterval(60000);), so the profile is still locked for a minute.

Makoto-san, is there a way to cancel the wait?

Comment 26

[Jorg K (CEST = GMT+2)]

Wow, bad typos: This implements a shutdown listener and the wait loop will exit.

Comment 27

[Makoto Kato [:m_kato]]

Comment on attachment 9015312 [details] [diff] [review]
1257058-shutdown-crash.patch

Review of attachment 9015312 [details] [diff] [review]:
-----------------------------------------------------------------

Since this patch adds observer per IMAP protocol, I think that it is better to add separated observer class for memory and etc.

And, although I think that AddObserver and RemoveObserver work on main thread only, is nsImapProtocol::Initialize always on main thread?

Comment 28

[Jorg K (CEST = GMT+2)]

Thank you.

(In reply to Makoto Kato [:m_kato] from comment #27)
> Since this patch adds observer per IMAP protocol, I think that it is better
> to add separated observer class for memory and etc.
I don't quite understand. You mean:

class nsImapProtocolObserver: public nsIObserver ...

nsImapProtocol::Initialize() {

  ...
  m_observer = new nsImapProtocolObserver();

Then in the loop:

  !m_observer->GetShutdown();

Why is that preferred?

I'd have to check your "main thread" question.

Back to my question from comment #25: Is there a way to cancel the wait?

Comment 29

[Makoto Kato [:m_kato]]

(In reply to Jorg K (GMT+2) from comment #28)
> Thank you.
> 
> (In reply to Makoto Kato [:m_kato] from comment #27)
> > Since this patch adds observer per IMAP protocol, I think that it is better
> > to add separated observer class for memory and etc.
> I don't quite understand. You mean:
> 
> class nsImapProtocolObserver: public nsIObserver ...

Yes, I mean that

StaticRefPtr<nsImapProtocolObserver> sShutdownObserver;
bool sShuttingDown = false;

...

nsImapProtocol::Initialize()
{
  ...
  if (!sShutdownObserver) {
    sShutdownObserver = new nsImapProtocolObserver();
    ...
  }
  ...
}
...

And sShutdownObserver  is set to nullptr when receiving shutdown observe.


> I'd have to check your "main thread" question.
> 
> Back to my question from comment #25: Is there a way to cancel the wait?

ReentrantMonitorAutoEnter.Notify() ?  TellThreadToDie is helper to die UI.  So I guess that when shutting down, we might have to call TellThreadToDie on shutdown observer.

Comment 30

[Jorg K (CEST = GMT+2)]

Comment on attachment 9015312 [details] [diff] [review]
1257058-shutdown-crash.patch

OK, let's give this another round. I tried
  m_passwordReadyMonitor.Notify()
yesterday and that crashed :-( - I don't remember the details now.

Comment 31

[Jorg K (CEST = GMT+2)]

Attachment: 1257058-shutdown-crash2.patch - WIP

Doesn't compile, I picked some bits and pieces from TimelineConsumers.cpp/h.

Comment 32

[Jorg K (CEST = GMT+2)]

Attachment: 1257058-shutdown-crash2.patch - WIP (v2)

Missing include :-( - Compiles but doesn't link:

0:13.64 lld-link.exe: error: undefined symbol: ??0nsImapProtocolObserver@@QEAA@XZ
 0:13.64 >>> referenced by c:\mozilla-source\comm-central\comm\mailnews\imap\src\nsImapProtocol.cpp:616
 0:13.64 >>>               ..\..\comm\mailnews\imap\src\nsImapProtocol.obj:(?Initialize@nsImapProtocol@@UEAA?AW4nsresult@@PEAVnsIImapHostSessionList@@PEAVnsIImapIncomingServer@@@Z)

Comment 33

[Jorg K (CEST = GMT+2)]

Attachment: 1257058-shutdown-crash2.patch - v3

Like this? Compiles and links now ;-) I don't think it works though, at least it behaves differently to the first version.

Looks like having a global 'sShuttingDown' caused the link error. However, I think we don't even need that "shutting down" variable now since we could just test 'sShutdownObserver' since the shutdown clears that.

Comment 34

[Jorg K (CEST = GMT+2)]

Actually, no, both versions work equally badly. After the minute has expired, I see:

Hit MOZ_CRASH(NSS_Shutdown failed) at c:/mozilla-source/comm-central/xpcom/build/XPCOMInit.cpp:1048
#01: XREMain::XRE_main (c:\mozilla-source\comm-central\toolkit\xre\nsAppRunner.cpp:4950)
#02: XRE_main (c:\mozilla-source\comm-central\toolkit\xre\nsAppRunner.cpp:5014)
#03: NS_internal_main (c:\mozilla-source\comm-central\comm\mail\app\nsMailApp.cpp:311)
#04: wmain (c:\mozilla-source\comm-central\toolkit\xre\nsWindowsWMain.cpp:143)
#05: __scrt_common_main_seh (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:283)
#06: BaseThreadInitThunk[C:\Windows\System32\KERNEL32.DLL +0x13034]
#07: RtlUserThreadStart[C:\Windows\SYSTEM32\ntdll.dll +0x71461]

Or maybe that's because we're still waiting too long and the shutdown still fails on timeout which is the problem we came to solve.

Comment 35

[Jorg K (CEST = GMT+2)]

Attachment: 1257058-shutdown-crash.patch (v1b)

I returned back to the previous approach where the protocol is also the listener. This way we can call NotifyAll() and that indeed ends the wait and gets us to the crash mentioned in comment #34 immediately, which on one hand is a good thing.

Now we have to work out why this crashes with
  MOZ_CRASH("NSS_Shutdown failed");

The comment above says:
// If you're seeing this crash and/or warning, some NSS resources are
// still in use (see bugs 1417680 and 1230312).

So I think this patch does roughly what we need, only that the XPCOM shutdown fails when it comes to shutting down NSS. I have no idea what resources we might still be locking.

Comment 36

[Makoto Kato [:m_kato]]

Although comment 33 approach is better, if it doesn't work well, you should use comment #35 approach.

Hmm, I guess that nss returns SEC_ERROR_BUSY. Becasue kImapSleepTime is too long??

Comment 37

[Jorg K (CEST = GMT+2)]

Thanks. The single observer option (comment #33) can't do this
  ReentrantMonitorAutoEnter mon(m_passwordReadyMonitor);
which seems to be needed to stop the wait.

I have to look into the crash in debug mode. In opt builds, there is no MOZ_CRASH() so the patch here would already "fix" the shutdown crash.

Comment 38

[Jorg K (CEST = GMT+2)]

Attachment: 1257058-shutdown-crash.patch (v1b) - rebased

I need to get back to this.

Comment 39

[Jorg K (CEST = GMT+2)]

The NSS_Shutdown() in ShutdownXPCOM() returns -1 (SECFailure), that doesn't really tell us why this is failing. Reducing kImapSleepTime from 60000 to 6000 (6 seconds instead of 60) as per comment #36 makes no difference.

Dana, you authored the MOZ_ASSERT() here
https://hg.mozilla.org/mozilla-central/rev/faa9d965f89e#l1.21
so I how can I find you what's stopping NSS from shutting down cleanly?

Other than that, the patch does what I want.

Comment 40

[Jorg K (CEST = GMT+2)]

I've done a bit more debugging, we fail since secmod_PrivateModuleCount==2 here:
https://searchfox.org/mozilla-central/rev/7c848ac7630df5baf1314b0c03e015683599efb9/security/nss/lib/pk11wrap/pk11util.c#91

Also, previously the call to STAN_Shutdown() called here
https://searchfox.org/mozilla-central/rev/efc0d9172cb6a5849c6c4fc0f19d7fd5a2da9643/security/nss/lib/nss/nssinit.c#1160
also failed.

In summary, something still appears to be busy, but I don't know what that something is and how to make it not busy.

Comment 41

[Jorg K (CEST = GMT+2)]

After looking at this for a while, I think this needs a completely different approach. New patch coming.

Comment 42

[Jorg K (CEST = GMT+2)]

Attachment: 1257058-fix-password-wait.patch

Looking at this again, the problem is this:

There is a wait loop what waits for a password to arrive asynchronously. One of the tests is/was m_password.IsEmpty(), so if the password is empty, it will keep looping, and cause a shutdown crash when TB is closed.

I think this logic is quite wrong. The problem arises in the user enters no password and simply clicks OK, see comment #0.

So if the user clicks OK and *empty* password *was indeed* entered, no point really to keep looping since the prompt is gone and nothing better will ever arrive.

It makes much more sense to break the loop when a password was entered, empty or not. If it was empty, and of course the authentication fails, the user is prompted again, and they can press cancel.

This seems like a very straight forward fix to a logic error which causes 1000 crashes a day :-(

Comment 43

[Magnus Melin [:mkmelin]]

Comment on attachment 9022735 [details] [diff] [review]
1257058-fix-password-wait.patch

Review of attachment 9022735 [details] [diff] [review]:
-----------------------------------------------------------------

Great detective work! r=mkmelin

Comment 44

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/9f8543c1794e
Distinguish 'empty password' from no password received to avoid shutdown crash. r=mkmelin

Comment 45

[Jorg K (CEST = GMT+2)]

Beta (TB 64 beta 2):
https://hg.mozilla.org/releases/comm-beta/rev/f9d7095faf28964f57252e818c13f725b15a9bee

Comment 46

[Ulf Zibis]

Thanks for finding the cause for this bug!
I would be happy, if the fix would be back-ported to the current version of TB.

Question:
Given a IMAP Server which regularly temporarily is in dysfunction or doesn't respond. In this case the user is prompted to enter a password in loop. I guess, if the user enters OK with empty password, the correct and before saved password is deleted, so if the IMAP server is responding correctly again, the connection fails due to the deleted password. I think, this is not desirable. Wouldn't it be better, the prompt would warn the user about the consequence acknowledging an empty password with OK?

Another fix could be to cancel the loop from the shutdown handler to avoid the crash.

Comment 47

[Jorg K (CEST = GMT+2)]

We intend to backport this once we got feedback from beta users. You will be able to try it in TB 65 beta due for release in early December. Or try a Daily build now (currently suspended, but they'll be back any day now).

The prompt in question is only issued if no password was saved. If a password was saved, you'll get the Retry/Enter new/Cancel prompt. I don't know why any user would OK an empty password, or even worse, click the "Use password manager" box. Surely they should click cancel if they don't want to enter a password.

> Another fix could be to cancel the loop from the shutdown handler to avoid the crash.
I tried that without success. However, as I wrote above, I don't think keeping the wait going if the user clicked OK on an empty password is not the correct approach since it will just loop forever. Why would empty and invalid (say they typed an x) be treated differently?

Comment 48

[Ulf Zibis]

(In reply to Jorg K (GMT+2) from comment #47)
Hi Jork, thanks for your quick answer.

I fully agree with your point of view.

If in the Retry/Enter new/Cancel prompt the user enters Cancel, I think, it would be reasonable to always double the wait time for the next prompt, so there would leave a chance, that the connection will be established some time later, if the IMAP server responds correctly again.
What you think?

Comment 49

[Wayne Mery (:wsmwk)]

(In reply to Jorg K (GMT+1) from comment #47)
> ...
> The prompt in question is only issued if no password was saved. If a
> password was saved, you'll get the Retry/Enter new/Cancel prompt. I don't
> know why any user would OK an empty password, or even worse, click the "Use
> password manager" box. Surely they should click cancel if they don't want to
> enter a password.

And yet, the crash signature has more than three times the number of installs crashing than the next highest ranked crash signature.  So some perhaps odd user behavior is driving this to significant levels.

> We intend to backport this once we got feedback from beta users. You will be
> able to try it in TB 65 beta due for release in early December. Or try a
> Daily build now (currently suspended, but they'll be back any day now).

I suggest you uplift this for 60.3.1.  The patch is simple, I've tested several variations of password prompting with 64.0b2, and there are no problem reports thus far even though 64.0b2 is out only a few days.

Full disclosure, I request this for the reason above but mainly because we have not definitive crash-stats data 
- this bug/crash sig doesn't have have a crash signature linked to it for beta 64 or 63, but I suspect it is is shutdownhang | nsThread::Shutdown | nsThreadManager::Shutdown
- shutdownhang | nsThread::Shutdown | nsThreadManager::Shutdown doesn't show any signs of improvement

So in this case, unless we wait on beta 2-3 weeks more, I think release channel is the only place we're going to clearly see how effective the patch is.

Comment 50

[Jorg K (CEST = GMT+2)]

(In reply to Wayne Mery (:wsmwk) from comment #49)
> I suggest you uplift this for 60.3.1.
I'll never say "no" to an uplift request ;-)

When do you want to ship 60.3.1? I did two builds today (the second one to "haste makes waste"), but I can do a third one.

Comment 51

[Jorg K (CEST = GMT+2)]

TB 60.3.1 ESR:
https://hg.mozilla.org/releases/comm-esr60/rev/f207024cf5d9b4c7b293b5c1db6608f949eab6ef

Comment 52

[Wayne Mery (:wsmwk)]

Thanks!

This topcrash is gone in 60.3.1

Comment 53

[Wayne Mery (:wsmwk)]

Also confirmed gone in 64.0 beta 2 (buildid 20181108115952) for signature shutdownhang | nsThread::Shutdown | nsThreadManager::Shutdown 
https://crash-stats.mozilla.com/signature/?product=Thunderbird&build_id=20181108115952&signature=shutdownhang%20%7C%20nsThread%3A%3AShutdown%20%7C%20nsThreadManager%3A%3AShutdown&date=%3E%3D2018-11-03T02%3A24%3A46.000Z&date=%3C2018-11-16T23%3A24%3A46.000Z#comments

Comment 54

[Wayne Mery (:wsmwk)]

Jorg

If the graphs below are to believed, we have a sharp rise in the last two days for both channels. A bit early to say, but it looks like most of that rate increase is going to be from 60.3.3.

(There was a significant drop in mid-November in beta [1] and release [2] crashes.)

[1] beta https://crash-stats.mozilla.com/signature/?product=Thunderbird&release_channel=beta&signature=shutdownhang%20%7C%20nsThread%3A%3AShutdown%20%7C%20nsThreadManager%3A%3AShutdown&date=%3E%3D2018-09-08T18%3A31%3A40.000Z&date=%3C2018-12-08T18%3A31%3A40.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=install_time&_sort=-date&page=1#graphs

[2] release https://crash-stats.mozilla.com/signature/?product=Thunderbird&release_channel=release&signature=shutdownhang%20%7C%20nsThread%3A%3AShutdown%20%7C%20nsThreadManager%3A%3AShutdown&date=%3E%3D2018-09-08T18%3A31%3A40.000Z&date=%3C2018-12-08T18%3A31%3A40.000Z#graphs