2050347 - EWS/NTLM on on-prem Exchange intermittently shows global login-failed dialog while sync continues; NTLM re-auth is not coordinated on the real POST channel

Status: RESOLVED FIXED

(MailNews Core :: Networking: Exchange, defect, P1)

Description

[Miha Markič]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:152.0) Gecko/20100101 Firefox/152.0

Steps to reproduce:

I connect to custom hosted Exchange server and get a flurry of dialog boxes telling me authentication failed. Note that I have many folders within my account and I assume thunderbird issues requests in parallel which causes this issue.

It's this bug https://bugzilla.mozilla.org/show_bug.cgi?id=2035812 but it seems that it wasn't fixed.

Actual results:

Plenty of dialog boxes: "Login to server MYSERVER with username MYUSERNAME failed."

This is this bug https://bugzilla.mozilla.org/show_bug.cgi?id=2035812 but it seems that it wasn't fixed.

Expected results:

I shouldn't see login failures.

Comment 1

[Eleanor Dicharry]

I think I'm able to reproduce this with our NTLM test account. I set up and configure the account successfully, but after I exit Thunderbird and come back, all attempts to authenticate with the server fail.

Comment 2

[Gregor Lichtner :glichtner]

I can confirm that the current Daily build of Thunderbird doesn't fix the original issue for me (I'm the author of the original bug report https://bugzilla.mozilla.org/show_bug.cgi?id=2035812 ).

Comment 3

[Brendan Abolivier [:babolivier]]

It seems like the fix from bug 2035812 didn't help (which I've also seen reported privately by someone else); I'll take another look (hopefully) next week.

Comment 4

[Brendan Abolivier [:babolivier]]

Attachment: Bug 2050347 - Also refresh the password of Exchange incoming servers when the auth method is NTLM. r=#thunderbird-back-end-reviewers

Comment 5

[Brendan Abolivier [:babolivier]]

I've found a possible cause for this issue; I've written a WIP fix for it and triggered builds that include this patch. Gregor and Miha, would you be able to test the one for your OS and see if it works for you?

Linux: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/FxORpaLXQ1KqxC6rosRNAA/runs/0/artifacts/public/build/target.tar.xz
macOS: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/VytyR9DoT_iFctk9jym-TQ/runs/0/artifacts/public/build/target.dmg
Windows: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/YsTfqpNGQQmrKb7_bo65gw/runs/0/artifacts/public/build/target.zip

Comment 6

[Gregor Lichtner :glichtner]

(In reply to Brendan Abolivier [:babolivier] from comment #5)

I've found a possible cause for this issue; I've written a WIP fix for it and triggered builds that include this patch. Gregor and Miha, would you be able to test the one for your OS and see if it works for you?

Linux: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/FxORpaLXQ1KqxC6rosRNAA/runs/0/artifacts/public/build/target.tar.xz
macOS: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/VytyR9DoT_iFctk9jym-TQ/runs/0/artifacts/public/build/target.dmg
Windows: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/YsTfqpNGQQmrKb7_bo65gw/runs/0/artifacts/public/build/target.zip

Thanks for your efforts! Unfortunately the error persists for me (Windows version of the ci-build). Let me know if it would be of help if I provide any logs.

Comment 7

[Miha Markič]

#metoo for Linux

Comment 8

[Brendan Abolivier [:babolivier]]

(In reply to Gregor Lichtner :glichtner from comment #6)

Thanks for your efforts! Unfortunately the error persists for me (Windows version of the ci-build). Let me know if it would be of help if I provide any logs.

Thanks for your offer Gregor! I'd be interested if you could send me some network logs, both with and without your patch from bug 2035812, so I can see what happens differently from the network's perspective. To generate network logs:

• Open Thunderbird's devtools with Ctrl+Shift+I (it might ask you to confirm that you want to allow the devtools to "connect" to your Thunderbird instance)
• Switch to the network tabs
• Perform operations in Thunderbird that would typically lead to the authentication issue you're seeing, e.g. synchronising a folder (if possible perform the same operations in both captures)
• Click on the gear icon in the top right corner of the devtools > Save All As HAR

If you can then send me the HAR file for each capture to brendan[at]thunderbird.net, that might help my investigation progress - there might be a clue in there regarding what we're doing differently than Necko's built-in HTTP error handling (which is tricky to use in Thunderbird for the reasons I've mentioned in the previous bug) 🙂

If you'd rather send me these files via encrypted email, my PGP key should be available on the main keyservers with the fingerprint A605 085C 29B3 6F8D 6CBE 40F4 CDCA 42A3 7FDF 8A72.

Comment 9

[Brendan Abolivier [:babolivier]]

Marking this bug as leave-open, since the problem persists but the password fix should still land.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/ab5c94cd7446
Also refresh the password of Exchange incoming servers when the auth method is NTLM. r=jtracey

Comment 11

[Brendan Abolivier [:babolivier]]

Attachment: Bug 2050347 - Check if the error handling line is open before sending an Exchange request. r=#thunderbird-back-end-reviewers

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/0084bda39413
Check if the error handling line is open before sending an Exchange request. r=edicharry

Comment 13

[Gregor Lichtner :glichtner]

Attachment: moz_http: build the Response from the channel the request finished on, not the pre-auth-retry channel

Comment 14

[Gregor Lichtner :glichtner]

Attachment: EWS: let Necko complete NTLM in-band on the request channel using stored credentials

Comment 15

[Gregor Lichtner :glichtner]

I'm still seeing the spurious credential prompts with NTLM on current trunk (local build of comm-central from 2026-07-21, which includes both landed patches). I dug into a MOZ_LOG capture (nsHttp:5,protocol_shared::*:5,moz_http::*:5) and found two separate issues. With the two attached patches applied, Thunderbird works against my on-premises Exchange server with NTLM auth (Windows 11 client) without a single prompt, at the default operation queue concurrency.

Disclosure: I used Claude Fable to help with the log analysis and to write these patches. They are running in my local build.

Issue 1 (independent of NTLM): moz_http reads the response from the wrong channel after Necko replaces it internally.

RequestBuilder::send() builds its Response from the nsIHttpChannel it originally created. But HTTP auth retries go through nsHttpChannel::RedirectToNewChannelForAuthRetry(), which creates a replacement channel. AsyncChannelOpener's stream listener receives that final channel in OnStopRequest(aRequest, ...) but ignores it. So when an authentication exchange succeeds, the listener gets the 200 response body, while Response::status() still reports the original channel's 401 — and the EWS layer treats a successful operation as an auth failure. From my log (trimmed):

nsHttpChannelAuthProvider::ProcessAuthentication [code=401]
nsHttpNTLMAuth::GenerateCredentials                     <- type 1
nsHttpChannel::RedirectToNewChannelForAuthRetry
(replacement channel: 401 + NTLM challenge)
nsHttpNTLMAuth::GenerateCredentials                     <- type 3
nsHttpChannel::RedirectToNewChannelForAuthRetry
nsHttpChannel::ProcessResponse [httpStatus=200]         <- auth succeeded
[...send_request] Response received for request ...
    (status 401): SyncFolderHierarchy                   <- stale status!
[...operation_sender] handling early failure: failed to authenticate

Patch 0001 captures the request's channel in BufferingStreamListener::OnStopRequest() and builds the Response from it (falling back to the original channel). This can bite any moz_http consumer — status and headers are wrong whenever Necko swaps the channel.

Issue 2: the out-of-band NTLM flow authenticates the wrong connection.

NTLM authenticates a TCP connection, not a session. ntlm::authenticate() does the handshake with hand-built Authorization headers on channels drawn from the shared connection pool, so the type-3 message can go out on a different connection than the one that issued the challenge — and even on success, only that one pooled connection is authenticated. Other queue runners still race into 401s, which I believe is why the intermittent prompts survived the previous fixes (unless the deployment persists auth state via a cookie).

Patch 0002 lets Necko complete the handshake in-band, on the channel that carries the actual request:

• For NTLM accounts, EWS requests get notification callbacks implementing nsIAuthPrompt2/nsIAuthPromptProvider that supply the incoming server's stored credentials (splitting domain\user if needed). They never prompt: on PREVIOUS_FAILED they decline, so the existing frontend logic stays in charge of re-prompting.
• Bodies are set via nsIUploadChannel2::ExplicitSetUploadStream(), so method and body survive the auth-retry replacement channels.
• After the user enters a new password, the NTLM path retries the real request in-band instead of the out-of-band validation flow, which can
  spuriously fail with a correct password when its requests land on different pooled connections.

The existing out-of-band flow stays as a fallback for the failure path. With both patches, each pooled connection authenticates itself transparently on first use, so parallel queue runners are safe and no cookie persistence is needed.

Comment 16

[pacman]

I can confirm this issue is still present in Thunderbird 153.0 ESR on Linux Mint.

Environment:

• Thunderbird 153.0 ESR
• Linux Mint
• Microsoft Exchange Server 2019 (on-premises)
• EWS
• NTLM authentication

Observed behavior:

• Account setup succeeds.
• Folder hierarchy is downloaded.
• Sending mail works.
• Receiving mail works.
• Message bodies can be opened.

However Thunderbird continuously displays:

"Login to exchange.example.com with username username user.name failed."

The login dialog immediately reappears after pressing either Retry or Cancel, making Thunderbird unusable.

Despite the login dialog, the account continues to synchronize and send/receive mail successfully.

I verified that authentication itself is correct:

• curl --ntlm against the EWS endpoint authenticates successfully.
• Evolution EWS works with the same server, credentials and NTLM authentication.

As a workaround I had to manually remove the EWS account from Thunderbird's prefs.js because the endless login dialog prevented access to the Account Settings.
The username configured in the account was "Example User", but the login dialog repeatedly reported a failure for username "user.name".

Comment 17

[pacman]

(In reply to pacman from comment #16)

I can confirm this issue is still present in Thunderbird 153.0 ESR on Linux Mint.

Environment:

• Thunderbird 153.0 ESR
• Linux Mint
• Microsoft Exchange Server 2019 (on-premises)
• EWS
• NTLM authentication

Observed behavior:

• Account setup succeeds.
• Folder hierarchy is downloaded.
• Sending mail works.
• Receiving mail works.
• Message bodies can be opened.

However Thunderbird continuously displays:

"Login to server exchange.example.com with username user.name failed."

The login dialog immediately reappears after pressing either Retry or Cancel, making Thunderbird unusable.

Despite the login dialog, the account continues to synchronize and send/receive mail successfully.

I verified that authentication itself is correct:

• curl --ntlm against the EWS endpoint authenticates successfully.
• Evolution EWS works with the same server, credentials and NTLM authentication.

As a workaround I had to manually remove the EWS account from Thunderbird's prefs.js because the endless login dialog prevented access to the Account Settings.
The username configured in the account was "Example User", but the login dialog repeatedly reported a failure for username "user.name".

Comment 18

[Brendan Abolivier [:babolivier]]

Attachment: Bug 2050347 - Defer to Necko's implementation of HTTP auth for Exchange requests. r=#thunderbird-back-end-reviewers

Comment 19

[Brendan Abolivier [:babolivier]]

This patch should finally fix NTLM auth with Exchange, thanks a ton everyone for helping with troubleshooting and investigating this issue!

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/a04353d1b298
Defer to Necko's implementation of HTTP auth for Exchange requests. r=edicharry