1601671 - websocket connects to DIRECT instead of Proxy-HTTP/HTTPS with SystemProxySettings

Status: VERIFIED FIXED

(Core :: Networking: WebSockets, defect, P1)

Description

[stephane.g]

Attachment: bug-proxy-websocket.txt

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36

Steps to reproduce:

On Windows 10.

• Launch Fiddler (Proxy)
• Be sure Proxy settings in Firefox are "Use System Proxy Settings"
• Go here https://websocket.org/echo.html
• Click Connect

Actual results:

Connection is OK but it doesn't go through proxy, it's a DIRECT connection (so it's a bug).

Expected results:

Connection should go into the proxy.

I tried with a PAC file and Manual proxy, there's no bug. Only with the "Use System proxy Settings" is buggy

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Comment 2

[Kershaw Chang [:kershaw]]

Hi Reporter,

We might need a full http log, so we can try to figure out the reason. Could you try to capture the log again?
Please also add proxy:5 to MOZ_LOG. Thanks.

Comment 3

[stephane.g]

Attachment: websocket-bug.moz_log

Full log

Comment 4

[stephane.g]

You can find both lines in the log :

"Use proxy from system settings: PROXY 127.0.0.1:8888" => https
"Use proxy from system settings: DIRECT" => websocket

Comment 5

[Kershaw Chang [:kershaw]]

Junior, could you take a look? Thanks.

Comment 6

[Junior [inactive]]

Michal, do we not proxy websocket under http on purpose?

Comment 7

[Junior [inactive]]

But we do proxy websocket under http for PAC/manual. I guess we need to fix it.

Comment 8

[Michal Novotny [:michal]]

(In reply to Junior [:junior] from comment #6)

Michal, do we not proxy websocket under http on purpose?

No, websocket should prefer SOCKS if available, then CONNECT via HTTP proxy and then DIRECT.

Comment 9

[Junior [inactive]]

For system proxy, Bug 1177909 makes websocket relies on RESOLVE_PREFER_HTTPS_PROXY set only.
But websocket sets both RESOLVE_PREFER_SOCKS_PROXY and RESOLVE_PREFER_HTTPS_PROXY after Bug 1577862.

Hence, we should make sure if both are set, SOCKS has more priority but we're able to fall back to HTTPS.

Comment 10

[Junior [inactive]]

To move forward, we need to bring flags to getProxyForURI and let classes who implement nsISystemProxySettings deal with those flags
https://searchfox.org/mozilla-central/rev/923eec8d2fb8078ebc7a33a9e1ce73eac01f7446/netwerk/base/nsISystemProxySettings.idl#38-41

Comment 11

[Junior [inactive]]

Mark P1 since it's regressed from 71. Let's try to fixed in 73 and uplift to 72.

Comment 12

[Junior [inactive]]

Attachment: Bug 1601671 - make websocket support proxying on SOCKS/HTTPS, r=michal

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7f813996c32a
make websocket support proxying on SOCKS/HTTPS, r=michal

Comment 14

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/7f813996c32a

Comment 16

[stephane.g]

The bug is still present in 73.0.1

Comment 17

[Junior [inactive]]

To be clear, current implementation prefers SOCKS proxy > HTTPS proxy > DIRECT.
"Use proxy from system settings: PROXY 127.0.0.1:8888" in Comment 4 indicates it's HTTP proxy, not HTTPS

Comment 18

[stephane.g]

You misread Comment 4, it's HTTPS

The implementation is ok but you may misunderstood the bug.
It comes from "Use System Proxy Settings" which doesn't detect HTTPS proxy (and maybe socks but I don't have one) for Websockets.
Check it with Fiddler proxy and http://www.websocket.org/echo.html
It only works if I enter manually the proxy in Firefox Settings with HTTP and HTTPS.
Also, Fiddler only set HTTP and HTTPS proxy in OS (no Socks) and unfortunately even the actual implementation, the connection is still DIRECT with the proxy via system settings.

Comment 19

[Michal Novotny [:michal]]

I can confirm that the bug is still present in 74.0b6 and I can reproduce it on Linux. When I set system proxy using gsettings as follows, neither ws nor wss uses the proxy:

gsettings set org.gnome.system.proxy mode 'manual'
gsettings set org.gnome.system.proxy.http port '3128'
gsettings set org.gnome.system.proxy.http host '192.168.192.31'
gsettings set org.gnome.system.proxy.https port '3128'
gsettings set org.gnome.system.proxy.https host '192.168.192.31'

Comment 20

[Arrigo Marchiori]

Confirmed also on Firefox 73.0.1 as currently available on Ubuntu 18.04, when using system environment variables.

$ env | grep prox
https_proxy=http://myproxy:3128
http_proxy=http://myproxy:3128
ftp_proxy=http://myproxy:3128

FWIW I am working under KDE, whose "System Settings" are set to "No Proxy".

Comment 21

[Arrigo Marchiori]

Maybe the problem (for Unix) could be in file firefox/toolkit/system/unixproxy/nsUnixSystemProxySettings.cpp

Method nsresult nsUnixSystemProxySettings::GetProxyForURI(nsIURI* aURI, nsACString& aResult) there calls nsIURI::GetScheme() to obtain the scheme and passes it to the other methods that query either GConf of the environment variables to obtain the correspondent proxy server.

Is it possible that websockets URL return "wss" as scheme, and therefore the other methods do not return the http proxy, but look for a (nonexistant) wss proxy instead?

Comment 22

[Junior [inactive]]

I don't have a Linux device (and not sure why it fails) at this moment, but I can reproduce in windows.

Here's why we fail in win:
We kick a proxy resolution request to PAC thread since we might hang on main thread in win
https://searchfox.org/mozilla-central/rev/202a285024f174c2d2bf2152d9cba90a03723eab/toolkit/system/windowsproxy/nsWindowsSystemProxySettings.cpp#42

Therefore, the special handling for WSS is discarded and we resolve from system proxy with scheme wss here
https://searchfox.org/mozilla-central/rev/202a285024f174c2d2bf2152d9cba90a03723eab/netwerk/base/nsPACMan.cpp#764-768

Comment 23

[Junior [inactive]]

(In reply to Arrigo Marchiori from comment #21)

Maybe the problem (for Unix) could be in file firefox/toolkit/system/unixproxy/nsUnixSystemProxySettings.cpp

Method nsresult nsUnixSystemProxySettings::GetProxyForURI(nsIURI* aURI, nsACString& aResult) there calls nsIURI::GetScheme() to obtain the scheme and passes it to the other methods that query either GConf of the environment variables to obtain the correspondent proxy server.

Is it possible that websockets URL return "wss" as scheme, and therefore the other methods do not return the http proxy, but look for a (nonexistant) wss proxy instead?

Thanks for jumping in.
If we pass wss, SOCKS is the only possible proxy. Hence we try wss first and https next. Last comment shows why we failed.

Comment 24

[Junior [inactive]]

Attachment: Bug 1601671 - let system proxy in PACMan resolves HTTPS for websocket, r=michal

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/20e918639082
let system proxy in PACMan resolves HTTPS for websocket, r=michal

Comment 26

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/mozilla-central/rev/20e918639082

Comment 27

[abodenlosz]

Confirmed the issue with Nightly 73.0a1 (2019-12-05) on Windows 10 x64.
Verified using Firefox 73.0 and 77.0 on Windows 10 x64. Updating the flag to verified.

Comment 28

[linus]

I am facing exactly this problem right now with Firefox 85.0.1 for Ubuntu canonical - 1.0
Unfortunately I do not know how to proide a log file as provided by the reporter.

Comment 29

[linus]

[Tracking Requested - why for this release]:

Comment 30

[linus]

Hm, this might also be a global problem of my Ubuntu Xenial 16.04, as I am having the same issue with Chromium where using the system proxy configuration is the only option...

Comment 31

[linus]

...indeed. Fixed this now. Sorry for the fuss.

If anybody else finds this:
"When setting up the proxy config in the Network Settings panel, leave the socks proxy blank."
https://askubuntu.com/questions/890274/slack-desktop-client-on-16-04-behind-proxy-server