Open Bug 1536787 Opened 5 years ago Updated 2 months ago

Whatsapp web doesn't work behind an NTLM proxy accessible with default credentials, ws downgrade reauthenticates to the proxy, channel treats it as an auth failure

Categories

(Core :: Networking: Proxy, defect, P2)

Product:
Core
Component:
Networking: Proxy
Version:
65 Branch
Type:
defect

Tracking

()

People

(Reporter: kronos333, Unassigned)

References

Details

(Keywords: regression, Whiteboard: [necko-triaged][ntlm][http-conn])

User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:64.0) Gecko/20100101 Firefox/64.0

Steps to reproduce:

Set a proxy, connect to https://web.whatsapp.com/

Actual results:

Firefox doesn't load whatsapp QR code, last version working correctly with proxy is 64.0.2

Expected results:

Firefox should load whatsapp QR code correctly.

This works for me using Firefox 66.0.1 on windows 10 and a http/https proxy (Fiddler) without authentication.
Does it work if you launch Firefox in the Firefox safemode: Menu button/Help/Restart with Add-ons disabled ?

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0 (20190325095153)

I've tested this report on Windows 10 using the latest Nightly and Fx release build. When I set the proxy and connect to https://web.whatsapp.com/ I can see the QR code displayed. Can you please re-test using the latest Nightly or Firefox release build and report back the result?

Component: Untriaged → Networking
Flags: needinfo?(kronos333)
Product: Firefox → Core

My OS is Win7x64
Intranet proxy is 10.xx.xx.xx on port 8080
I try to clean all: registry, C:\Program Files, AppData (Local, LocalLow and Roaming) making a fresh installation, the result is the same.

Firefox 64.0.2 is still working perfectly with and without proxy
Firefox 66.0.1 is working only with "No proxy", safemode don't solve the problem

Flags: needinfo?(kronos333)

Are you able to install applications on your system because you seem to be in a corporate environment ?
In that case I would point you to a Mozilla regression tool that lets you find the exact change in Firefox that caused this.

Are you using a "manual proxy configuration" http/https proxy or the "use system proxy settings" or with "automated proxy configuration URL" ?
There was a change in recent Firefox versions related to "system settings" and "automated proxy" settings. The "no proxy" field in the configuration is now followed with this 2 modes and that was not the case with previous Firefox versions.

I tried also with an admin local account, out from domain, I tried both "use system proxy settings" and "manual proxy configuration".
The strange point is: why if I downgrade from 66.0.1 to 64.0.2 start to work perfectly?
I'm sorry, but now my temporarily solution is to stop Firefox updates and to use 64.0.2 version.

Has STR: --- → yes

Please provide a log according: https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging#Start_logging_using_command_line_arguments

Best would be to have a log from 64 (the working version) and a newer version (probably 65) that manifests the problem, so we could compare.

Thanks.

Flags: needinfo?(kronos333)

Logging done, how to attach these two files?

Log_64.0.2.zip 0.7M
Log_66.0.1.zip 6.7M

Flags: needinfo?(kronos333)

Will be deleted after 24 hours

https://filesharing24.com/d/CxR

Downloaded, thanks.

kronos333, can you please find out the URL of the QR code image (in the old firefox version)? I don't have whatsup by hand, thanks. Right click, then show image info or copy image link should tell you. Thanks.

Flags: needinfo?(kronos333)

ntlm@68(esr) -> P1

Assignee: nobody → honzab.moz
Status: UNCONFIRMED → NEW
Ever confirmed: true
Priority: -- → P1
Whiteboard: [necko-triaged][ntlm]

OK, I can see the difference now. kronos333, you forgot to mention one important thing - the proxy requires an ntlm auth. We fail the proxy auth for a request for a websocket.

Going to look into it.

NTLM is working, this is more an authentication restart issue in our code... This is an indirect regression from bug 1434137 (uncover of a bug somewhere else).

2019-03-27 14:46:28.273000 UTC - [Parent 1440: Socket Thread]: I/nsHttp Http2Session::ProcessWaitingWebsockets 0000000020D82000 Re-queuing websocket as h1 due to mPeerAllowsWebsockets=false

What I exactly see in the log:

  • there is a web socket initiated request for https://web.whatsapp.com/ws
  • we successfully authenticate to the proxy with NTLM (get 200 CONNECTED)
  • the end point server is h2
  • we restart the ws request and enforce h1
  • we create a new connection to the proxy
  • the (somewhat fragile) logic in the http channel believes we failed to authenticate the proxy with the default credentials and kills the cached entry
2019-03-27 14:46:28.317000 UTC - [Parent 1440: Main Thread]: D/nsHttp nsHttpChannelAuthProvider::GetCredentialsForChallenge [this=000000001DD3B6A0 channel=0000000020E1B718 proxyAuth=1 challenges=NTLM]
2019-03-27 14:46:28.317000 UTC - [Parent 1440: Main Thread]: D/nsHttp nsHttpAuthCache::GetAuthEntryForDomain [key=http://lelapomi.telecomitalia.local:8080 realm=]
2019-03-27 14:46:28.317000 UTC - [Parent 1440: Main Thread]: V/nsHttp nsHttpNTLMAuth::ChallengeReceived [ss=0000000000000000 cs=000000001E2F4E80]
2019-03-27 14:46:28.317000 UTC - [Parent 1440: Main Thread]: V/nsHttp Force use of generic ntlm auth module: 0
2019-03-27 14:46:28.317000 UTC - [Parent 1440: Main Thread]: D/nsHttp   identity invalid = 1
2019-03-27 14:46:28.317000 UTC - [Parent 1440: Main Thread]: D/nsHttp nsHttpChannel::CloseStickyConnection this=0000000020E1B000
2019-03-27 14:46:28.317000 UTC - [Parent 1440: Main Thread]: V/nsHttp nsHttpConnection::DontReuse 0000000020E85800 spdysession=0000000000000000
2019-03-27 14:46:28.317000 UTC - [Parent 1440: Main Thread]: D/nsHttp nsHttpChannel::ConnectionRestartable this=0000000020E1B000, restartable=0
2019-03-27 14:46:28.317000 UTC - [Parent 1440: Main Thread]: D/nsHttp   clearing bad auth cache entry
2019-03-27 14:46:28.317000 UTC - [Parent 1440: Main Thread]: D/nsHttp Destroying nsHttpAuthNode @000000001FAEE8C0

(the auth node is created by the very first channel, using default credentials.)

  • we ask for credentials with a prompt
  • you canceled the prompt, which canceled the request

The channel must be notified the proxy connection is re-attempted and not because of authentication failure. This is going to be a bit tricky.

Blocks: 1434137
Flags: needinfo?(kronos333)
Priority: P1 → P2
Summary: Whatsapp web doesn't work behind a proxy → Whatsapp web doesn't work behind an NTLM proxy, ws downgrade
Summary: Whatsapp web doesn't work behind an NTLM proxy, ws downgrade → Whatsapp web doesn't work behind an NTLM proxy accessible with default credentials, ws downgrade reauthenticates to the proxy, channel treats it as an auth failure

I don't canceled the prompt, somewhere are stored, Firefox do not prompt for user and password, I can connect any website without to specify user and password. The strange is the difference between the two versions of Firefox, the old success, the new fails.

(In reply to kronos333 from comment #14)

I don't canceled the prompt,

Right, I could see an attempt in the log, but it failed sooner than we managed to open the prompt (also an interesting bug, btw):

2019-03-27 14:46:28.317000 UTC - [Parent 1440: Main Thread]: D/nsHttp nsHttpChannelAuthProvider::PrepareForAuthentication [this=000000001DD3B6A0 channel=0000000020E1B718]
2019-03-27 14:46:28.317000 UTC - [Parent 1440: Main Thread]: D/nsHttp nsHttpChannelAuthProvider::GetAuthenticator [this=000000001DD3B6A0 channel=0000000020E1B718]
2019-03-27 14:46:28.317000 UTC - [Parent 1440: Main Thread]: D/nsHttp nsHttpChannelAuthProvider::GetAuthenticator [this=000000001DD3B6A0 channel=0000000020E1B718]
2019-03-27 14:46:28.317000 UTC - [Parent 1440: Main Thread]: D/nsHttp nsHttpChannelAuthProvider::GetCredentialsForChallenge [this=000000001DD3B6A0 channel=0000000020E1B718 proxyAuth=1 challenges=NTLM]
2019-03-27 14:46:28.317000 UTC - [Parent 1440: Main Thread]: D/nsHttp nsHttpAuthCache::GetAuthEntryForDomain [key=http://lelapomi.telecomitalia.local:8080 realm=]
2019-03-27 14:46:28.317000 UTC - [Parent 1440: Main Thread]: V/nsHttp nsHttpNTLMAuth::ChallengeReceived [ss=0000000000000000 cs=000000001E2F4E80]
2019-03-27 14:46:28.317000 UTC - [Parent 1440: Main Thread]: V/nsHttp Force use of generic ntlm auth module: 0
2019-03-27 14:46:28.317000 UTC - [Parent 1440: Main Thread]: D/nsHttp   identity invalid = 1
2019-03-27 14:46:28.317000 UTC - [Parent 1440: Main Thread]: D/nsHttp nsHttpChannel::CloseStickyConnection this=0000000020E1B000
2019-03-27 14:46:28.317000 UTC - [Parent 1440: Main Thread]: V/nsHttp nsHttpConnection::DontReuse 0000000020E85800 spdysession=0000000000000000
2019-03-27 14:46:28.317000 UTC - [Parent 1440: Main Thread]: D/nsHttp nsHttpChannel::ConnectionRestartable this=0000000020E1B000, restartable=0
2019-03-27 14:46:28.317000 UTC - [Parent 1440: Main Thread]: D/nsHttp   clearing bad auth cache entry
2019-03-27 14:46:28.317000 UTC - [Parent 1440: Main Thread]: D/nsHttp Destroying nsHttpAuthNode @000000001FAEE8C0
2019-03-27 14:46:28.317000 UTC - [Parent 1440: Main Thread]: D/nsHttp nsHttpChannelAuthProvider::PromptForIdentity [this=000000001DD3B6A0 channel=0000000020E1B718]
2019-03-27 14:46:28.320000 UTC - [Parent 1440: Main Thread]: D/nsHttp nsHttpChannelAuthProvider::GetAuthenticator [this=000000001DD3B6A0 channel=0000000020E1B718]
2019-03-27 14:46:28.320000 UTC - [Parent 1440: Main Thread]: D/nsHttp nsHttpChannelAuthProvider::GetCredentialsForChallenge [this=000000001DD3B6A0 channel=0000000020E1B718 proxyAuth=1 challenges=Basic realm="Griffon - Navigazione Internet"]
2019-03-27 14:46:28.320000 UTC - [Parent 1440: Main Thread]: D/nsHttp nsHttpChannel::CloseStickyConnection this=0000000020E1B000
2019-03-27 14:46:28.320000 UTC - [Parent 1440: Main Thread]: D/nsHttp nsHttpChannel::ConnectionRestartable this=0000000020E1B000, restartable=1
2019-03-27 14:46:28.320000 UTC - [Parent 1440: Main Thread]: D/nsHttp nsHttpChannelAuthProvider::PromptForIdentity [this=000000001DD3B6A0 channel=0000000020E1B718]
2019-03-27 14:46:28.320000 UTC - [Parent 1440: Main Thread]: D/nsHttp unable to authenticate
2019-03-27 14:46:28.320000 UTC - [Parent 1440: Main Thread]: D/nsHttp ProcessAuthentication failed [rv=80004002]

somewhere are stored,

no, it works differently. for ntlm proxies we let Windows authenticate automatically.

Firefox do not prompt for user and password, I can connect any website without to specify user and password. The strange is the difference between the two versions of Firefox, the old success, the new fails.

I believe I explained the difference.

I understood now, thanks for explanation, is a Windows authentication, NTLM was unknown for me, I read now something about NTLM authorization proxy.
Thanks

Whatsapp web is loaded correctly in Firefox 64.0.2 version because don't use NTLM method?
I will keep 64.0.2 version, or for Whatsapp web connections I will use another browser.

Will you solve this problem in one of the next releases of Firefox?

(In reply to kronos333 from comment #18)

Will you solve this problem in one of the next releases of Firefox?

Maybe in the current cycle, for 68. But no guarantee.

Ok, no hurry, many thanks for your answer.

To solve this we have to force the code we invoke at [1] to happen always after we have successfully connected the proxy (got 200 Connected) - the transaction has to tell the channel somehow. Dropping the continuation state makes the code at [2] use default creds again (continuationState == nullptr.)

Adding folks working on the socket process isolation to make them aware and maybe chime in. This would require some re-architecture of this ancient code (to improve communication between connection, transaction and channel) or to have a special message sent from the transaction to the channel (a part #2 or a followup to this bug.)

Leaving the [ntlm] tag as this bug disallows use of the default credentials.

[1] https://searchfox.org/mozilla-central/rev/ddd1679c0534f7ddf36cafddd17b710c4fefe3c4/netwerk/protocol/http/nsHttpChannelAuthProvider.cpp#455
[2] https://searchfox.org/mozilla-central/rev/ddd1679c0534f7ddf36cafddd17b710c4fefe3c4/netwerk/protocol/http/nsHttpNTLMAuth.cpp#186

Whiteboard: [necko-triaged][ntlm] → [necko-triaged][ntlm][http-conn]
Status: NEW → ASSIGNED

In Firefox 67.0.2 the problem still persists.

Same problem with Firefox 71.0.1 (64 bit) on MacOS High Sierra 10.13.6. However I've got the same with Chrome and Safari.....

(In reply to am from comment #23)

Same problem with Firefox 71.0.1 (64 bit) on MacOS High Sierra 10.13.6. However I've got the same with Chrome and Safari.....
Firefox 72.0.1 sorry

I still have the same issue with Firefox 74 (64 bit) on MacOS High Sierra 10.15.4 (19E266). Chrome and Safari works well with system proxy configured.

Bugbug thinks this bug is a regression, but please revert this change in case of error.

Keywords: regression
Assignee: honzab.moz → nobody
Status: ASSIGNED → NEW

Assigning to myself to look into NTLM issue.

Assignee: nobody → valentin.gosu
Severity: normal → S3

Moving bug to Core/Networking: Proxy.

Assignee: valentin.gosu → nobody
Component: Networking → Networking: Proxy
You need to log in before you can comment on or make changes to this bug.