1869297 - Shutdown crash due to "Empty Trash on Exit" for inactive / inaccessible account (password related for many crash reporters)

Status: RESOLVED WORKSFORME

(Thunderbird :: General, defect, P2)

Description

[Arthur K. (he/him)]

EDIT: See comment 11 for STR.

No STR here. This bug is being opened to further investigate instances where the fix in bug 1864747 does not fully fix all shutdown hangs on "Take 2" build of 115.5.2 that lead to a crash.

Crash example: https://crash-stats.mozilla.org/report/index/0bb63e0e-93b1-4ef3-975b-c35a70231211

Perf profile: https://share.firefox.dev/3NmpPJk

Feel free to rename the bug to better suit the pursuit.

Comment 1

[Magnus Melin [:mkmelin]]

Unfortunately I don't see what's wrong there.

Comment 2

[Arthur K. (he/him)]

(In reply to Magnus Melin [:mkmelin] from comment #1)

Unfortunately I don't see what's wrong there.

Boy, don't I feel special all of a sudden. =\

Is there anything I can tweak on my Profiler Settings that might yield even an iota more info? I think I have nearly every option enabled which tries to capture anything and everything. Could it be some setting in my profile?

I know a while back that someone asked me if it shuts down quickly without a hang or crash when I go into Offline mode and exit TB. I remember doing that experiment and it shuts down fully within 1-1.5 seconds. I re-tested that theory and it does shutdown within about a second. Any correlation to that?

Comment 3

[Magnus Melin [:mkmelin]]

What kind of accounts do you have? And if you set one/all to not connect, does it make a difference if no connection was active?
Or potentially something similar for calendars and/or address books? I'm thinking some connection left open and things related.

Comment 4

[Arthur K. (he/him)]

(In reply to Magnus Melin [:mkmelin] from comment #3)

What kind of accounts do you have? And if you set one/all to not connect, does it make a difference if no connection was active?
Or potentially something similar for calendars and/or address books? I'm thinking some connection left open and things related.

And truly the most bizarre thing about this bug is it only happens on my home PC. My office PC running Portable 115.5.1 never has issues. Nor does my Portable TB 115.5.1 running of my USB stick either.

On my home PC where it happens, I have 3 gmail accounts, 1 Exchange account using OWL add-on, 1 inactive Comcast account and my old work account which is also inactive. By inactive I mean that they are set to not go out and check a server or grab mail. My 3 gmail accounts are all OAuth2. My primary gmail account has a calendar as does my Exchange account.

So would you like me to try disabling all account temporarily and seeing if it shuts down normally? Same thing with calendars as well?

Comment 5

[Magnus Melin [:mkmelin]]

Right. Though if you have an owl account, try Help | Troubleshoot mode first.

Comment 6

[Arthur K. (he/him)]

(In reply to Magnus Melin [:mkmelin] from comment #1)

Unfortunately I don't see what's wrong there.

The problem with the perf profile is that it's probably not capturing the actual moment of the crash. Is there some way to capture the entirety from the time I do a File > Exit action up until the crash window appears? I have no way to capture the full sequence since TB shuts down and takes the profiler with it but the process is still running Task Manager.

Comment 7

[Wayne Mery (:wsmwk)]

https://profiler.firefox.com/docs/?_gl=1*11me5k3*_ga*MTMxODk5MDkzOC4xNjk5MzIzMDY4*_ga_2VC139B3XV*MTcwMjM0OTkwNi4xMzAuMC4xNzAyMzQ5OTA2LjAuMC4w#/./guide-startup-shutdown?id=shutdown

Comment 8

[Arthur K. (he/him)]

(In reply to Magnus Melin [:mkmelin] from comment #3)

What kind of accounts do you have? And if you set one/all to not connect, does it make a difference if no connection was active? Or potentially something similar for calendars and/or address books? I'm thinking some connection left open and things related.

I did figure out why Comcast Empty Trash On Exit still works. Apparently as an ex-customer, they still let you use your old email so for me it's not been deactivated. // I figured out the problem but it kind of doesn't makes total sense why it was happening. Some history: I changed jobs earlier this year and before I left I saved/copied over 24 years worth of emails but didn't delete my email profile (IMAP) in TB for this old job. I simply unchecked "Check for new messages at startup", "Check for new messages every" and "Allow immediate server notifications when new messages arrive" under Server Settings think I was good.

I then thought about what you said Magnus: "I'm thinking some connection left open and things related." So, I unchecked "Check for new messages at startup", "Check for new messages every", "Allow immediate server notifications when new messages arrive" and "Empty trash on exit" for all accounts and restarted TB.

After doing this, a File > Exit closed TB instantly without crash. The plot thickens. One by one on each functional account, I re-enabled "Check for new messages at startup", "Check for new messages every", "Allow immediate server notifications when new messages arrive" and "Empty trash on exit." A File > Exit closed TB instantly without crash.

That is, until I got to my inactive old work account. I still kept unchecked "Check for new messages at startup", "Check for new messages every" and "Allow immediate server notifications when new messages arrive" but left enabled "Empty trash on exit" and wouldn't you know it THAT was the culprit.

So, all that being said, it begs the question for me: Why does enabling "Empty trash on exit" for my inactive Comcast account not crash TB but leaving it enabled on my inactive old work account crash TB?

Does leaving "Empty trash on exit" enabled still send out a command to empty trash an inactive account and leave TB stuck in an error state because the inactive account in question doesn't accept the credentials anymore but TB doesn't know that and just falls over and crashes? If that's the case, we need to bail out if "Empty trash on exit" is enabled and just shut down rather than be stuck in a state where the server didn't accept the credentials (because we no longer have right to access the email server) and TB doesn't handle the exception.

Comment 9

[Magnus Melin [:mkmelin]]

Great detective work!
I wasn't able to reproduce though.

The code seems like it should be prepared to handle shutdown. Code is around here: https://searchfox.org/comm-central/rev/28bdb11caabaa6ff12f724a9d8e437c9e7b4cc90/mailnews/base/src/nsMsgAccountManager.cpp#1568,1608

If you're willing to run a special debug build I could create one that prints out what's happening.

Comment 10

[Arthur K. (he/him)]

(In reply to Magnus Melin [:mkmelin] from comment #9)

Great detective work!
I wasn't able to reproduce though.

The code seems like it should be prepared to handle shutdown. Code is around here: https://searchfox.org/comm-central/rev/28bdb11caabaa6ff12f724a9d8e437c9e7b4cc90/mailnews/base/src/nsMsgAccountManager.cpp#1568,1608

If you're willing to run a special debug build I could create one that prints out what's happening.

Yes, let's go for it. Would it just be a build I can install over my existing 115.2.2 build 2? Let me know what you'd need from me.

Comment 11

[Arthur K. (he/him)]

(In reply to Magnus Melin [:mkmelin] from comment #9)

I wasn't able to reproduce though.

I think this would be repro STR:

1. Using an IMAP account, change the password so your TB account cannot access it (meaning, TB would normally throw a password login prompt message)
2. Edit the Settings for this account and uncheck "Check for new messages at startup", "Check for new messages every" and "Allow immediate server notifications when new messages arrive" but leave "Empty trash on exit" enabled
3. Restart TB and then when TB has settled, do a File > Exit action

TB should be trying to do an Empty Trash action on the IMAP account with the changed password but can't because our credentials are not valid to do the action but we're not being notified. I think that's the gist of it.

Comment 12

[Wayne Mery (:wsmwk)]

Still #1 crash

Comment 13

[Arthur K. (he/him)]

(In reply to Wayne Mery (:wsmwk) from comment #12)

Still #1 crash

Is it possible to make TB disable "Empty Trash on Exit" if the other three options ("Check for new messages at startup", "Check for new messages every" and "Allow immediate server notifications when new messages arrive") are also disabled? Or, at the very least, if "Check for new messages at startup", "Check for new messages every" and "Allow immediate server notifications when new messages arrive" are disabled it doesn't bother doing an "Empty Trash on Exit" action?

Comment 14

[Wayne Mery (:wsmwk)]

We wouldn't normally take that approach. For one thing, we'd have to remember the user's prior settings and restore them when the crisis is over.

It's not clear from the bug's crash graph that Thunderbird's crashing has increased, but using soccoro it is clear https://crash-stats.mozilla.org/signature/?signature=shutdownhang%20%7C%20mozilla%3A%3ASpinEventLoopUntil%20%7C%20nsThread%3A%3AShutdown%20%7C%20nsThreadManager%3A%3AShutdownNonMainThreads&date=%3E%3D2023-07-29T04%3A20%3A00.000Z&date=%3C2024-01-29T04%3A20%3A00.000Z#graphs

Comment 15

[Wayne Mery (:wsmwk)]

If you're willing to run a special debug build I could create one that prints out what's happening.

Magnus can you create the try build?

Comment 16

[Magnus Melin [:mkmelin]]

Finally got around to this. Arthur, please get a try build from here: https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=1b16dfec3cccd84b3b6bdcfc3a00b23aedd2013e&selectedTaskRun=cZfE-pk3RG6yXGpH2_SR-g.0
Windows installer: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/cZfE-pk3RG6yXGpH2_SR-g/runs/0/artifacts/public/build/install/sea/target.installer.exe
It will print some "xxxmagnus ... " debug info to console while shutting down. IIRC on windows you have to start with -console to see that.

Comment 17

[Arthur K. (he/him)]

Hmm, somehow I edited Comment 8 and unintentionally blew out all what I had written. Does anyone have that content in one of their email replies from me? I'd like to put it back the way it was.

Comment 18

[Arthur K. (he/him)]

(In reply to Magnus Melin [:mkmelin] from comment #16)

Finally got around to this. Arthur, please get a try build from here: https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=1b16dfec3cccd84b3b6bdcfc3a00b23aedd2013e&selectedTaskRun=cZfE-pk3RG6yXGpH2_SR-g.0
Windows installer: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/cZfE-pk3RG6yXGpH2_SR-g/runs/0/artifacts/public/build/install/sea/target.installer.exe
It will print some "xxxmagnus ... " debug info to console while shutting down. IIRC on windows you have to start with -console to see that.

I grabbed this file and will try it on my home PC. Will this be a 115.x.x version?

Comment 19

[Arthur K. (he/him)]

(In reply to Wayne Mery (:wsmwk) from comment #14)

We wouldn't normally take that approach. For one thing, we'd have to remember the user's prior settings and restore them when the crisis is over.

So, if it would not be a good idea to do so, could TB be made to just bail out if Empty Trash On Exit is enabled for an account that's no longer active (and, as in my case, it's just hanging and eventually crashing due to an unseen error) and under the hood the server response is an error (assuming here it's an invalid credentials response from the server)?

Comment 20

[Wayne Mery (:wsmwk)]

(In reply to Arthur K. (he/him) from comment #17)

Hmm, somehow I edited Comment 8 and unintentionally blew out all what I had written. Does anyone have that content in one of their email replies from me? I'd like to put it back the way it was.

I combined your one sentence with the original content.

Comment 21

[Wayne Mery (:wsmwk)]

(In reply to Arthur K. (he/him) from comment #19)

(In reply to Wayne Mery (:wsmwk) from comment #14)

We wouldn't normally take that approach. For one thing, we'd have to remember the user's prior settings and restore them when the crisis is over.

So, if it would not be a good idea to do so, could TB be made to just bail out if Empty Trash On Exit is enabled for an account that's no longer active (and, as in my case, it's just hanging and eventually crashing due to an unseen error) and under the hood the server response is an error (assuming here it's an invalid credentials response from the server)?

Perhaps a better, or parallel, solution is Bug 689067 - Ability to disable an IMAP account without losing e-mail?

Orthogonal - Support is rife with user reports of "I can't access my account". In general we probably should be more intelligent about what we do/what we offer the user when an account server cannot be contacted - we rely too much on the user making a decision but at the same time likely don't provide enough options and advice that would allow the user to decide the best course of action. (not a great bug query https://mzl.la/4bH7nFI )

Comment 22

[Magnus Melin [:mkmelin]]

Yes the try is a 115 build

Comment 23

[Arthur K. (he/him)]

Attachment: console-output.txt

Here's a crash report: https://crash-stats.mozilla.org/report/index/60a08492-5855-47e3-beb5-b0c4b0240221

Console output didn't show a whole lot though. Anything else you need me to do?

Comment 24

[Magnus Melin [:mkmelin]]

Please get the console output form this try build which has more debug printfs: https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=8cce43e937fa7cb520bb2d93456eb9536ed3bc7f&selectedTaskRun=OxQFonoGSRyzxbefkFvDlQ.0

Windows installer: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/OxQFonoGSRyzxbefkFvDlQ/runs/0/artifacts/public/build/install/sea/target.installer.exe

Comment 25

[Arthur K. (he/him)]

Attachment: console-output-try2.txt

Here's output from the newer binary.

Comment 26

[Arthur K. (he/him)]

(In reply to Magnus Melin [:mkmelin] from comment #24)

Please get the console output form this try build which has more debug printfs: https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=8cce43e937fa7cb520bb2d93456eb9536ed3bc7f&selectedTaskRun=OxQFonoGSRyzxbefkFvDlQ.0

Windows installer: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/OxQFonoGSRyzxbefkFvDlQ/runs/0/artifacts/public/build/install/sea/target.installer.exe

Was there anything of use in the console output?

Comment 27

[Wayne Mery (:wsmwk)]

(In reply to Arthur K. (he/him) from comment #25)

Created attachment 9387970 [details]
console-output-try2.txt

Here's output from the newer binary.

extract ...

waynemery@waynemery-mbp-16-2021 ~ % grep -v "loopCount" Downloads/bug1869297-attachment.txt
xxxmagnus nsMsgAccountManager::CleanupOnExit
xxxmagnus ...hostName=imap.gmail.com
xxxmagnus ...emptyTrashOnExit=1, cleanupInboxOnExit=0
xxxmagnus ...emptyTrashOnExit!
xxxmagnus ...emptyTrashOnExit; will wait for up to 5s
xxxmagnus ... /emptyTrashOnExit
xxxmagnus ...hostName=mail.vinakom.com
xxxmagnus ...hostName=imap.comcast.net
xxxmagnus ...hostName=imap.gmail.com
xxxmagnus ...emptyTrashOnExit=1, cleanupInboxOnExit=0
xxxmagnus ...emptyTrashOnExit!
xxxmagnus ...emptyTrashOnExit; will wait for up to 5s
xxxmagnus ... /emptyTrashOnExit
xxxmagnus ...hostName=smart mailboxes
xxxmagnus ... will CloseCachedConnections
xxxmagnus /nsMsgAccountManager::CleanupOnExit
RunWatchdog: Mainthread nested event loops during hang:
--- nsThread::Shutdown: IMAP

Comment 28

[Arthur K. (he/him)]

(In reply to Magnus Melin [:mkmelin] from comment #9)

Great detective work!
I wasn't able to reproduce though.

The code seems like it should be prepared to handle shutdown. Code is around here: https://searchfox.org/comm-central/rev/28bdb11caabaa6ff12f724a9d8e437c9e7b4cc90/mailnews/base/src/nsMsgAccountManager.cpp#1568,1608

If you're willing to run a special debug build I could create one that prints out what's happening.

Magnus,

Although you were not able to reproduce, can you reproduce using the following STR:

1. Using an existing and working IMAP account (i.e., gmail.com) for which the password is stored in TB, change the following options for it via the Account Settings > Server Settings page:

Server Settings:
a. [ ] Check for new messages at Start (unchecked / disabled)
b. [ ] Check for new messages every X minutes (unchecked / disabled)
c. [ ] Allow immediate server notifications when new messages arrive (unchecked / disabled)

When I delete a message:
(o) Move it to this folder [Trash on [e-mail])

Message Storage
a. [ ] Clean up ("Expunge") Inbox on exit (unchecked / disabled)
b. [X] Empty Trash on Exit (checked / enabled)

2. Close TB
3. Via web browser, log into this known working IMAP account and change the password. Keep your new password handy, obviously.
4. Start TB once more. Since you've disabled checking for messages in Server Settings, it shouldn't query this account.
5. File > Exit TB. TB should try to empty trash on this account but since the password stored in TB is different than the password that was changed in Step 3, it should get stuck and crash.

Comment 29

[Magnus Melin [:mkmelin]]

I did find the cause, kind of. Just not clear what to do about it. Stay tuned.

Comment 31

[Wayne Mery (:wsmwk)]

(In reply to Magnus Melin [:mkmelin] from comment #29)

I did find the cause, kind of.

Magnus can you elaborate?

Comment 32

[Wayne Mery (:wsmwk)]

This is our #1 crash for 115 by a big margin. Ranking for 128 TBD.

I am sure this is all OS. Most likely for Mac is its #1 crash shutdownhang | mozilla::SpinEventLoopUntil<T> | nsThread::Shutdown | nsThreadManager::ShutdownNonMainThreads. Perhaps related to Bug 1524247? Thunderbird Shutdown crash/hang due to endless wait loop when no password is entered | WaitForSingleObjectEx | WaitForSingleObject | _PR_MD_WAIT_CV | _PR_WaitCondVar | PR_WaitCondVar | mozilla::CondVar::Wait | nsEventQueue::GetEvent

(In reply to Wayne Mery (:wsmwk) from comment #31)

(In reply to Magnus Melin [:mkmelin] from comment #29)

I did find the cause, kind of.

Magnus can you elaborate?

Magnus indicated in chat "Not easily, need to track some stuff down. "

Comment 33

[Wayne Mery (:wsmwk)]

Arthur, are you still seeing this crash?

Comment 34

[Arthur K. (he/him)]

I am not but only because I have turned off "Empty Trash on Exit" on my old disabled work account. I'm almost certain that if I re-enable that setting that it will crash again. I am keen to try it out on 128 when it goes live in the upcoming weeks to see if the crash persists or if the signature has changed.

If you want me to try to crash TB based on comment 8 stuff, let me know. I am on 115.13.0 on my home PC.

Comment 35

[gene smith]

I tried to duplicate Arthur's STR but observed something else. If I "disable" the account using his method but never visit a folder in the disabled account during the session, there is no attempt to delete and expunge messages in Trash folder at shutdown even if "delete trash on shutdown" is enabled. Unless a folder in the disabled account is visited, causing a login to the server to occur, there is no attempt made by TB to clean out trash on shutdown, i.e., the "deleteallmessages" URL on Trash for the disabled account never occurs so messagess are not marked \deleted and no expunge occurs. I verified this using an IMAP:4 log. I haven't researched why this is true.

So it appears that unless Arthur is visiting the disabled account before shutting down, there will be no attempt to delete trash messages, and whatever is causing the crash at shutdown is caused by something else.

I mostly run with a self-build daily build, and at shutdown, maybe half the time, I see a crash dump with the offer to run "gdb" to debug the problem. I typically just ignore this since it happens so much. So maybe Arthur is just seeing something random like this and it has nothing to do with "empty trash on shutdown"?

One other thing that I've mentioned before. This patch is still in daily: https://phabricator.services.mozilla.com/D192569. For me, with this present in daily, "empty trash on shutdown" usually doesn't occur since it causes an authentication failure. That patch was never merged into 115 so "empty trash on shutdown" works OK with 115 but is broken in daily. I don't know if that patch is in beta or if it is targeted for merge into 128.

Comment 36

[Arthur K. (he/him)]

Seems to be fixed. Tested and reported in bug 1704637 that setting Empty Trash on Exit using and old work account that I no longer have access to doesn't hang or crash. Tested with 128.3.2. Shall we close this bug?

Comment 37

[Wayne Mery (:wsmwk)]

Shall we close this bug?

Yes, thanks very much.
The crashes stop after 119.0b6 (beta)

Comment 38

[Francesco]

As observed in comment #35, the "Empty Trash on Exit" function was partly disabled, and apparently that fixes the crash. See bug 1896545.