1793599 - IMAP to fastmail fails after erasing ImapMail folder

Status: RESOLVED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[Kai Engert (:KaiE:)]

To reduce used disk space, I changed "sync & storage" settings for all IMAP accounts, and disabled "keep messages .. on this computer".

Then I quit Thunderbird, and deleted the ImapMail folder.
Is this an allowed cleanup action? Should Thunderbird handle this gracefully?

For my fastmail account, folder display is stuck at "Inbox" only.
The "progress" indicator in the mail tab is actively moving, but nothing happens.

(Thunderbird was able to re-download headers for all my additional IMAP accounts.)

I've used MOZ_LOG="IMAP:5" to get some initial tracing, which I'll attach.

For testing purposes, I've restored the contents of the ImapMail/imap.fastmail.com folder from a backup (which I took prior to deleting from the profile directory).

With that directory restored, accessing the fastmail account immediately works again.

Comment 1

[Kai Engert (:KaiE:)]

Attachment: fastmail-sync-fail.txt

Comment 2

[Kai Engert (:KaiE:)]

The error log claims that login failed.
This is unexpected, because the password is remembered by Thunderbird.
I use a primary password, but I'm entering it correctly at startup.

Comment 3

[Kai Engert (:KaiE:)]

SMTP is unaffected. I was able to compose an email from that stuck fastmail account. The email was sent, and copying to the sent folder stalls, too.

Comment 4

[Kai Engert (:KaiE:)]

FYI, I've reproduced this with 102.3.1 (and my existing profile)

Comment 5

[Kai Engert (:KaiE:)]

Gene what do you think? Should this work?

Comment 6

[Magnus Melin [:mkmelin]]

I don't know about Fastmail specifically, but nuking ImapMail will recreate it (fixed bug 1777937 a few months back). I didn't see any problems back then, but testing was on Dovecot.

Comment 7

[gene smith]

Kai, I don't have a fastmail account to test this with but fastmail uses Cyrus imap from what I understand. I do have local Cyrus imap server for testing and I don't see the problem after deleting the ImapMail directory.
Your log fragment show that and "EnsureExists" URL is occurring for Junk folder to make sure it exists. This appears to be the first URL ready to run and is causing the hang for fastmail account. The reason is because "EnsureExists" has no "Window" associated with it and here
https://searchfox.org/comm-central/rev/ead29136c954ba6aafe0adcf53734a54be401038/mailnews/imap/src/nsImapProtocol.cpp#8368
the password can't be obtain so the connection fails. On normal startup, URLs "select" and "discoverallbox" are first and have an associated window and don't have a problem getting the stored password.
I don't know why the EnsureExists URL is happening for Junk. Other URLs can do the same thing when they are "Windowless" like URL "folderstatus" that was noticed here: bug 1282182.
In bug 1282182 I attached a possible diff to fix the issue. It now has some bit-rot and now, looking at it some more, don't think it is a valid change.
I need to look at this some more. Edit: Looking again, I think it's OK, after some updates, as described in next comment 8.
Sorry for the delay in responding.

Comment 8

[gene smith]

Here's a try build for a possible fix: https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=ad0bfd35368604e39c8c95369810fb6eff26fe40 It's basically what I proposed in bug 1282182. If there is no window, instead of giving up it now checks if password manager has a password stored. If it does, the password is returned and used to authenticate. If not, an empty password and an error is returned and no prompt occurs, as is expected when there is no window to show it on.
Kai said in comment 2, the password is "remembered". So with the try build when the initial windowless "EnsureExists" occurs the stored password should be obtained directly from password manager and TB will continue on rather than just hanging.

However, I don't think this would fix the problem that Kai is seeing if passwords were not remembered/stored and had to be entered at startup. So I still need to understand why "EnsureExists" is occurring first.

Not sure if Kai can still duplicate the bug and then test with the try build. Anyhow, maybe Kai and/or Magnus could just take a look at the associated diff and see if it looks reasonable or know a better way to address this issue.

The issue of windowless URLs failing to authenticate has come up in a couple of other contexts such as the already mentioned bug 1282182. I first became aware of it in bug 1690093 where Ben Bucksch pointed out why this line exists: https://searchfox.org/comm-central/rev/ead29136c954ba6aafe0adcf53734a54be401038/mailnews/imap/src/nsImapProtocol.cpp#8368.
Bug 1690093 was also triggered by the "EnsureExists" URL but didn't cause a failure to start up since it wasn't the first URL to occur. The fix there added a "window" to one occurrence of EnsureExists but didn't add a window for where it's failing for Kai: see the 3rd parameter here https://searchfox.org/comm-central/rev/1149553854dc7cfac00639ef11fc3c79fa4d47d2/mailnews/imap/src/nsImapMailFolder.cpp#1047

Comment 9

[Kai Engert (:KaiE:)]

I can still reproduce the bug. However, I prefer testing with comm-esr102, because I'm experiencing the bug on a machine that should remain on 102.

I have applied your patch https://hg.mozilla.org/try-comm-central/rev/ad0bfd35368604e39c8c95369810fb6eff26fe40 to comm-esr102 - on opt of the fix from bug 1629669 to make your patch apply cleanly, and I'm testing my local 102 build.

Comment 10

[Kai Engert (:KaiE:)]

Gene, great work, your fix resolves my issue. Thank you!

Comment 11

[Kai Engert (:KaiE:)]

FWIW, the issue cannot simply be caused by cyrus imapd in general. I'm using that software on my own server that handles mail for kuix.de, and the same profile didn't have issues with my account(s) on my server.

Comment 12

[gene smith]

Glad it worked. It worked for me when I simulated the error you see but I was not 100% sure it would work in real world.

Some questions:
Is the log fragment you attached the full log for fastmail server? I.e., is there anything deleted from log fragment for fastmail from earlier time than where your log starts?
Is there anything special about the fastmail "Junk" folder that it is trying to ensure exists?
Have you tried doing the the delete of ImapMail with an earlier version, e.g., 91?
If you shutdown tb and go inside ImapMail and delete just the directory for fastmail and restart, does the EnsureExists occur first?
With tb running, if you delete the "Junk" folder and restart tb, does the problem not occur or does it just do "EnsureExists" on another folder?
Are you using mbox or maildir? I tried both and it didn't seem to matter. (I realize you are trying to get rid of offline storage but you still have to select one or the other.)
If you setup a new test profile and new account with only fastmail and do delete of ImapMail, does that also also do the EnsureExists first?

Sorry for all the questions. Maybe if I had a temporary fastmail account I could more easily duplicate the problem. If you can provide one, let me know the credentials. Otherwise, maybe I could ask Bron.

Comment 13

[gene smith]

Questions in comment 12 may be moot. I suspect it's because your "Junk" folder has a localized name. That would cause the URL EnsureExists since CreateStorageIfMissing invokes it:

https://searchfox.org/comm-central/rev/0b0768b7a4c022316bb6d09e84677d5f9ffe6535/mailnews/base/src/nsMsgUtils.cpp#845
https://searchfox.org/comm-central/rev/0b0768b7a4c022316bb6d09e84677d5f9ffe6535/mailnews/imap/src/nsImapMailFolder.cpp#1047

Not sure why this would be the first URL but maybe I can simulate it and find out...

Comment 14

[Kai Engert (:KaiE:)]

(In reply to gene smith from comment #12)

Is the log fragment you attached the full log for fastmail server? I.e., is there anything deleted from log fragment for fastmail from earlier time than where your log starts?

I'm pretty sure I didn't truncate or emit anything.

Is there anything special about the fastmail "Junk" folder that it is trying to ensure exists?

The folder is empty. Thunderbird is configured to disable adaptive junk controls. "move new junk messages" is configured to move them to the "spam" folder on the fastmail imap account.

Because I have spam messages in the "spam" folder, I assume that "spam" is the folder that fastmail has chosen, because the fastmail service is moving spam messages there.

I notice that on another computer, Thunderbird has the junk folder subscribed, but doesn't show it in the folder list!

I'm pretty sure that I never manually created this junk folder. Is it likely that Thunderbird had created it?

Have you tried doing the the delete of ImapMail with an earlier version, e.g., 91?

No, I never attempted this previously.

If you shutdown tb and go inside ImapMail and delete just the directory for fastmail and restart, does the EnsureExists occur first?
With tb running, if you delete the "Junk" folder and restart tb, does the problem not occur or does it just do "EnsureExists" on another folder?
If you setup a new test profile and new account with only fastmail and do delete of ImapMail, does that also also do the EnsureExists first?

I can try to run these experiments soon.

Are you using mbox or maildir? I tried both and it didn't seem to matter. (I realize you are trying to get rid of offline storage but you still have to select one or the other.)

I'm using whatever the default is. I'm not even sure about the difference.

Maybe if I had a temporary fastmail account I could more easily duplicate the problem.

That might be easiest as a first step, and I could attempt to trace for you, should your fresh account not reproduce the issue.

If you can provide one, let me know the credentials. Otherwise, maybe I could ask Bron.

I think you can very easily sign up for free for a 30 day trial on the fastmail dot com page.

Comment 15

[Kai Engert (:KaiE:)]

(In reply to gene smith from comment #13)

Questions in comment 12 may be moot. I suspect it's because your "Junk" folder has a localized name.

The web interface of fastmail shows the folder as "Junk".

I'm using an en-us thunderbird binary.
No localized folder names. Thunderbird displays it as "Junk".

Comment 16

[gene smith]

I'm using an en-us thunderbird binary.
No localized folder names. Thunderbird displays it as "Junk".

Ok, good to know. Maybe it's still going there for some other reason or I'm on the wrong trail completely.

Comment 17

[gene smith]

I've partially duplicate it (ensureExists first URL) when I create and mark as the junk folder a folder called "gds". After deleting ImapMail and restarting, a normal folder called "Junk" is created and "gds" remains marked with junk icon. However, since the o365 server is oauth2 based, the authentication doesn't fail and the only very minor problem is that unexpected "Junk" folder is created. So I think if I try this on a "Plain" password based system it will duplicate. Server vendor is N/A as I/we suspected. The oauth2 account is in Spanish (es) but apparently that doesn't matter since your account is en-US.

Comment 18

[gene smith]

Yes, it now duplicates with a plain password setup and I see the same hang reported in comment 0 and log info as reported in comment 1. Of course, this is without my proposed patch https://hg.mozilla.org/try-comm-central/rev/ad0bfd35368604e39c8c95369810fb6eff26fe40

STR for a plain password based account:

Notice which folder currently has the "junk" icon (e.g., folder Spam)
Create a folder called, e.g., gds
Configure account to use gds as junk folder
Delete folder Spam or Junk if they exist (maybe optional)
Shutdown TB
Delete profile directory ImapMail or just delete account specific directories and files under ImapMail
Start TB with imap logging, e.g. MOZ_LOG=IMAP:5,timestamp,sync MOZ_LOG_FILE=~/tblog ../obj-x86_64-pc-linux-gnu/dist/bin/thunderbird --allow-downgrade -p
Only Inbox is visible but no messages present and subfolders never appear as the should even after selecting Inbox
Imap log show basically the same thing as comment 1, failure to login. No other error indications.

Comment 19

[gene smith]

My proposed patch is attached next. It includes all the changes described in comment 8 and contained in the "try" diff https://hg.mozilla.org/try-comm-central/rev/ad0bfd35368604e39c8c95369810fb6eff26fe40 that Kai applied to his 102 system. However, as mentioned in comment 8, that change won't work when a user does not store the account/server password and instead enters it at the start of each session.

So the proposed patch includes other changes to handle the more general case where the first URL that occurs before any imap connections exists is "windowless" so it won't support a password prompt. If the current session for the server has never been authenticated and the URL doesn't have an associated window, the URL is queued, allowing a later URL that does have a window to run first and authenticate. Then after authentication, deferred URLs are pulled from the queue and allowed to run on the now authenticated server session.

With normal TB startup, a URL with a window, discoverallboxes, occurs first on a not yet authenticated session followed by URL select. The discoverallboxes runs first but the select is always queued until discoverallboxes authenticates (either from a stored password or from a password prompt). Then, if authentication is successful, select is pulled from the queue and runs followed by many other URLs.

The bug occurred when the first URL was windowless, such as the observed ensureExists URL. The patch adds a check for a window on the first URL to occur in a not yet authenticated session with zero imap connections. If the URL has no window and there is no locally cached password stored, the URL is queued. It's possible that more than one initial windowless URL may occur consecutively and are added to the queue (I haven't seen this). When the first URL with a window occurs, i.e., discoverallboxes, it will establish the imap connection and run and handle the authentication. Then the queued URLs will run in the now authenticated session along with an other URLs following discoverallboxes.

The check for a "locally cached password" mentioned above was added to pass the imap unit tests. Otherwise, it's not really needed for normal TB operation. In the imap unit tests, often the only URL is fetch, which is windowless. The fetch URL was getting queued and hanging the unit tests since no subsequent URL with a window occurs. Apparently the "fake imap server" pre-stores the password in the nsMsgIncomingServer object so, unlike during normal operation, the password is not obtained from the login/password manager, therefore the stored local password is not empty even though the session is not yet authenticated.

I haven't tried to determine exactly why URL ensureExists occurs at startup for "Junk" folder. I do see the folder "Junk" getting created unnecessarily when imap detects a special folder labeled as "Junk" but TB is configured to store junk emails in a folder with another name like "Spam". So you end up with a "Spam" and "Junk" folder. But this only occurs when you delete everything for the account inside ImapMail directory and restart TB. This may be a minor bug in itself needing some investigation.

Here a "try" build with this patch that looks clean: https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=f2cc35d69ac29a54e4eab27b6dc7183b92e83196

Here are some workarounds for this bug that can bring the system back without my patch. I haven't tried to determine exactly why these work but I suspect they trigger a URL with a window that frees up the system. The 2nd two are, obviously, much simpler.

#1:
Delete the ImapMail folder(s) and files for problem account and start tb.
With TB hung and just Inbox showing, click on Inbox
Create a new folder under Inbox called anything, e.g., Ibox
Nothing happens.
Now click on account name
Click back to Inbox and all the folders appear (but Ibox is not created which is OK)

#2
Selecting Inbox and compacting it shows all the folders! (Probably won't work for maildir since no right-click compact exists)

#3
Right-click subscribe on account, Cancel and collapse/expand the folders shows all the folders.

Comment 20

[gene smith]

Attachment: Bug 1793599 - Queue initial windowless imap URLs to avoid system hang at startup. r=benc

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/d3021f313bf2
Queue initial windowless imap URLs to avoid system hang at startup. r=benc

Comment 22

[Wayne Mery (:wsmwk)]

Comment on attachment 9297929 [details]
Bug 1793599 - Queue initial windowless imap URLs to avoid system hang at startup. r=benc

[Triage Comment]
Approved for beta (discussed on Matrix with mkmelin)

Comment 23

[Wayne Mery (:wsmwk)]

Comment on attachment 9297929 [details]
Bug 1793599 - Queue initial windowless imap URLs to avoid system hang at startup. r=benc

correcting flags - this is already on beta.

[Triage Comment]
Approved for esr102

Comment 24

[Rob Lemley [:rjl]]

Thunderbird 102.4.2:
https://hg.mozilla.org/releases/comm-esr102/rev/7b0bbd650be4