2063816 - Gmail IMAP unusable with Thunderbird 153 ESR: intermittently but very frequently gives authentication errors and messages not loading. “Authentication failure while connecting to server imap.gmail.com.”

Status: ASSIGNED

(MailNews Core :: Networking: IMAP, defect)

Description

[Lupusolus]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:153.0) Gecko/20100101 Firefox/153.0

Steps to reproduce:

1. Open Thunderbird with an existing Gmail account configured using OAuth2.
2. Start Thunderbird and open the Gmail account.
3. Try to download new messages or open an existing Gmail message.
4. Repeat the same test after restarting Thunderbird.

The issue occurs intermittently but very frequently on two separate Windows computers.

Actual results:

Thunderbird frequently shows the notification:
“Authentication failure while connecting to server imap.gmail.com.”

Gmail synchronization then stalls. New messages are not downloaded reliably, and when opening Gmail messages the message pane often remains completely blank/white.

Occasionally the connection suddenly succeeds and Thunderbird downloads all pending Gmail messages at once. Shortly afterwards, Gmail may stop working again.

The same problem occurs on two separate Windows computers with separate Thunderbird installations/profiles. It also occurred on different networks (home network, hotel Wi-Fi and mobile connection).

Other mail accounts in Thunderbird (e.g. GMX) continue to work normally.

The affected Gmail account works normally with Aqua Mail on Android and Gmail webmail.

Removing and re-adding the Gmail account on one affected computer did not solve the problem. Thunderbird Troubleshoot Mode did not solve it either.

The problem already occurred intermittently with Thunderbird 153.0.2 ESR and currently persists with 153.0.3 ESR. Before the upgrade from Thunderbird 140 ESR to the 153 ESR branch, this behavior was not observed.

A third computer was upgraded from Thunderbird 140 ESR to 153 ESR and its Gmail account currently continues to work normally, so the issue does not appear to affect every Gmail account or installation.

Expected results:

Gmail should authenticate reliably, synchronize new messages automatically, and load message contents immediately when a message is opened, as it did previously with Thunderbird 140 ESR.

Comment 1

[Wayne Mery (:wsmwk)]

So these accounts were already oauth authenticated to gmail before moving to 153?

Comment 2

[Lupusolus]

Yes. The affected Gmail accounts were already configured and authenticated with OAuth2 in Thunderbird before upgrading from 140 ESR to 153 ESR.

As part of troubleshooting, I have since removed and re-added the Gmail account on one of the affected computers (the desktop/Tower), including a fresh Google OAuth authentication. This did not resolve the issue; the same problems still occur.

Comment 5

[Magnus Melin [:mkmelin]]

Confirming based on multiple reports.

Comment 6

[Magnus Melin [:mkmelin]]

Bug 2064058 may be related, it will be hard to prove.

Comment 7

[Wayne Mery (:wsmwk)]

Attachment: Login to account xxxx failed.

FWIW, in 155 (not 153) on startup I occassionally started seeing "Login to account xxxx failed.". There should not have been a login failure, because this is an oauth account with unexpired oauth tokens stored in Thunderbird.

Comment 8

[Wayne Mery (:wsmwk)]

Lupusolus (or anyone else),

Can you get an imap log? https://wiki.mozilla.org/MailNews:Logging

Comment 9

[Wayne Mery (:wsmwk)]

Bug 2065428 reports same issue, and finds this only happens with 64bit Thunderbird. It should be easy for others to confirm - you should be able to install 32bit of the same TB version (into the same program director) and still use the same Thunderbird profile data.

Windows 32bit 153esr - https://download.mozilla.org/?product=thunderbird-153.1.0esr-SSL&os=win&lang=en-US

Comment 10

[Lupusolus]

Both affected Windows systems are running Thunderbird 153 ESR 64-bit.

Regarding the requested IMAP log: I currently can't provide one because I have temporarily removed the Gmail account from Thunderbird. The Gmail IMAP issue had become severe enough that it was also slowing down normal use of my other mail accounts, especially with Unified Folders enabled. I am currently forwarding Gmail messages to my primary mail account instead.

I would prefer not to reconfigure Gmail in my production setup just for logging at this point. If a log from my particular setup is still important after the logs already provided by other reporters, please let me know and I can reconsider.

Comment 11

[Worcester12345]

Any relationship between this bug and those affecting Yahoo mail IMAP accounts?

Comment 12

[Max Emig [:maxe]]

(In reply to Worcester12345 from comment #11)

Any relationship between this bug and those affecting Yahoo mail IMAP accounts?

I would suspect so, I have a murky idea of what is going on.

Comment 13

[Wayne Mery (:wsmwk)]

I could try to get a log tonight. Or perhaps Max has seen something in logs I previously provided that might explain both my issue and this bug?

See of any of https://mzl.la/4xZ5luS are related.

Comment 14

[Andrei [:sancus]]

I spent a little bit of time investigating this and I couldn't find any evidence or reproduction that indicated it's due to recent Gecko networking changes or anything like that, which was my initial hypothesis. I could be wrong, of course.

But in any case, you will see authentication failure if gmail times out while we're waiting for the XOAUTH2 response. Many people have noticed gmail latency issues of late so this could be a symptom of that.

And there are a lot of things in this code that make it unnecessarily difficult to debug:

• We collapse protocol and network errors with eachother and with authentication errors, so you can't identify which phase or step of auth failed from any reported error. And the actual errors themselves are misleading to the user. Bug 849540 and Bug 1400661.
• We hold RunLogonExclusxive, the IMAP auth lock, across the whole auth process so if it stalls waiting for network timeouts it stalls all IMAP connections for the entire account and the timeout is really long.
• The login process is not very well phased in general. There should probably be different timeouts for different phases, like DNS,TCP, AUTH XOAUTH2 send/response, etc.
• A single stalled auth should not affect other accounts or any other part of desktop, and any sorts of connection errors in one account shouldn't cause issues elsewhere...
• Retry logic is kind of haphazard and I don't think we actually retry at all on a timed out XOAUTH2 response. We should consistently retry regardless of which phase the failure happens in, and maybe not throw visible errors at the user on a single intermittent network failure
• I don't think there is any telemetry for IMAP connections or especially auth? None that I could find, anyway. So we cannot identify if any error type started more frequently occurring with a specific Thunderbird version, or if we are affected by external server, network, or other factors. That would also require having useful error types.

All of these things are in pretty old code and haven't been changed in a long time.

Comment 15

[Max Emig [:maxe]]

All of these things are in pretty old code and haven't been changed in a long time.

Well, there have been OAuth2 changes.

And no @Wayne, nothing conclusive yet, more logs are always good.

Comment 16

[Max Emig [:maxe]]

Attachment: Bug 2063816 - Improve IMAP recovery from transport failures and prevent post-authentication server setup from blocking logins on other connections. r=#thunderbird-back-end-reviewers

Comment 17

[Max Emig [:maxe]]

Can anyone affected try the build of my draft patch?

https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=87313ac7c90504f873453e7b5265198d7643f6db

Ended up mitigating the top ideas of what went wrong.

Comment 18

[Worcester12345]

(In reply to Max Emig [:maxe] from comment #17)

Can anyone affected try the build of my draft patch?

https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=87313ac7c90504f873453e7b5265198d7643f6db

Ended up mitigating the top ideas of what went wrong.

Just for Gmail, or also YahooMail?

Comment 19

[Max Emig [:maxe]]

(In reply to Worcester12345 from comment #18)

(In reply to Max Emig [:maxe] from comment #17)

Can anyone affected try the build of my draft patch?

https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=87313ac7c90504f873453e7b5265198d7643f6db

Ended up mitigating the top ideas of what went wrong.

Just for Gmail, or also YahooMail?

If you are talking about bug 2053842, possibly, nothing about it is Gmail-specific. Also, I could not find a full IMAP log for your failure, could you provide one?

Comment 20

[Wayne Mery (:wsmwk)]

(In reply to Max Emig [:maxe] from comment #17)

...
Ended up mitigating the top ideas of what went wrong.

What are those top ideas? And are there other possible symptoms not mentioned in the bug title? Are there other ideas not included in the patch?

Comment 21

[Wayne Mery (:wsmwk)]

(In reply to Max Emig [:maxe] from comment #17)

Can anyone affected try the build of my draft patch?

I've been using it since Tuesday but I'm afraid I don't have much to report. My only symptom has been password prompts on network changes which was only intermittent, plus I've not been changing much the last two days

Comment 22

[Wayne Mery (:wsmwk)]

(In reply to Max Emig [:maxe] from comment #17)

Can anyone affected try the build of my draft patch?

https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=87313ac7c90504f873453e7b5265198d7643f6db

herboraninvlammen?
arthemesia?
Rama?

• Windows binary at https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/FLJohtM5Rx-LmzFXkfZWzA/runs/0/artifacts/public/build/setup.exe
• Linux at https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/P0Jmq44lQYakZh8wCXQagA/runs/0/artifacts/public/build/target.tar.xz

Comment 23

[Max Emig [:maxe]]

(In reply to Wayne Mery (:wsmwk) from comment #22)

(In reply to Max Emig [:maxe] from comment #17)

Can anyone affected try the build of my draft patch?

https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=87313ac7c90504f873453e7b5265198d7643f6db

herboraninvlammen?
arthemesia?
Rama?

• Windows binary at https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/FLJohtM5Rx-LmzFXkfZWzA/runs/0/artifacts/public/build/setup.exe
• Linux at https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/P0Jmq44lQYakZh8wCXQagA/runs/0/artifacts/public/build/target.tar.xz

To be clear, these are the core ideas behind it:

• Do not misreport an OAuth2 transport failure, such as timing out while waiting for the XOAUTH2 response, as rejected authentication; preserve GetConnectionStatus().
• Release RunLogonExclusive() once OAuth2 authentication succeeds, moving ProcessAfterAuthenticated() outside the lock. Holding it while waiting for slow post-authentication commands prevents other IMAP connections for the same account from logging in, so one stalled connection can stall the entire account.
• Add a one-second delay before the existing RetryUrl() attempt after a transport failure.