1601871 - REGRESSION: Websocket connection behind HTTP proxy does not work

Status: RESOLVED FIXED

(Core :: Networking: Proxy, defect, P2)

Description

[Mathias Homann]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:71.0) Gecko/20100101 Firefox/71.0

Steps to reproduce:

I'm trying to use a website that uses websckets from behind a HTTP proxy

Steps to reproduce:

• set up Firefox 71 to use a HTTP proxy
• try websockets, for example the websockets test on https://www.websocket.org/echo.html

Actual results:

"it does not work"

Expected results:

It should work, it used to in FF 70 and before.

Possibly a regression of https://bugzilla.mozilla.org/show_bug.cgi?id=1177909 and/or https://bugzilla.mozilla.org/show_bug.cgi?id=1517782

Comment 1

[Mathias Homann]

additional info:

just having a http proxy configured and in use breaks it. I have put the html test page from https://www.websocket.org/echo.html on my local server, which is bypassed from going through the proxy, and that already breaks it. As soon as I turn the proxy OFF, it works.

Comment 2

[Mathias Homann]

another test: even if the websockets test page is on localhost and localhost is explicitely excluded in the proxy settings it is broken.

Comment 3

[alekiv]

Same there. After upgrade from 70.0.1 to 70.1 Websockets connections through proxy don't work.

Comment 4

[Dragana Damjanovic [:dragana]]

Can you please try to run
https://mozilla.github.io/mozregression/

Thank you!

Comment 5

[Mathias Homann]

sorry, can't, at least on linux, too many python version conflicts.

isn't there a python3 version? python 2 is end of life!

I'll try on windows.

Comment 6

[Mathias Homann]

ok so I've gone through the process with mozregression-gui on windows, now what?

here's the end of the output:

2019-12-07T18:08:52: INFO : Narrowed inbound regression window from [caa26030, bb918aa3] (3 builds) to [a417315a, bb918aa3] (2 builds) (~1 steps left)
2019-12-07T18:08:52: DEBUG : Starting merge handling...
2019-12-07T18:08:52: DEBUG : Using url: https://hg.mozilla.org/integration/autoland/json-pushes?changeset=bb918aa3c389d3e78e5cbefffbacdb0faefea6a0&full=1
2019-12-07T18:08:53: DEBUG : Found commit message:
Bug 1562664 - Replace XUL textbox usages with HTML inputs in editBookmarkPanel.inc.xul r=bgrins

Differential Revision: https://phabricator.services.mozilla.com/D36495

2019-12-07T18:08:53: DEBUG : Did not find a branch, checking all integration branches
2019-12-07T18:08:53: INFO : The bisection is done.
2019-12-07T18:08:53: INFO : Stopped

the full log file is here:
https://gist.github.com/lemmy04/ba9629dd4ac1ea19cff8cfc8d3daf775

Comment 7

[Alice0775 White]

from comment#6 full log
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=15ffd69c&tochange=8b8ce461

I suspect the culprit bug is a417315a75d54d0c89dba01b5fea9a9fe6e92ee8 Michal Novotny — Bug 1577862 - Websockets should prioritize SOCKS5 proxies over HTTP(S) proxies, r=valentin

Comment 8

[Mathias Homann]

Here's an interesting little detail which kinda goes with that suspicion:

when I turn on "use this proxy for everything" it tries to use the squid proxy on port 3128 for socks5 as well. and fails.
when I don't turn that on, and instead put the proper port in for the socks setup (1080), websockets work fine even with the proxy use enabled.

Comment 9

[Mathias Homann]

food for thought:

• it might be said that this is actually a "works as designed" case
• in that case, should "use this proxy for all protocols" really include socks as well? I don't know if there are any proxies that do support socks AS WELL as all the rest, but I do know that squid doesn't...

Comment 10

[Michal Novotny [:michal]]

(In reply to Mathias Homann from comment #8)

when I don't turn that on, and instead put the proper port in for the socks setup (1080), websockets work fine even with the proxy use enabled.

What did you have set as host:port for SOCKS when it didn't work? Bug 1577862 changed preferred proxy type for websockets. If you have no host and port specified for SOCKS proxy, HTTP proxy should be used.

Comment 11

[Mathias Homann]

that is my point: when you select the option "use this proxy for all protocols" it puts that proxy and port into the socks settings as well, which is plain wrong - it should not overwrite what is already there unless the field was left empty.
I don't know if there even are HTTP proxy services that support socks on the same port.

See attached image for clarification.

Comment 12

[Mathias Homann]

Attachment: composited image of two states of proxy settings

Comment 13

[Michal Novotny [:michal]]

Then the code works as expected. IMO the option "Use this proxy server for all protocols" doesn't make any sense and bug 1577862 made this just obvious.

Comment 14

[Mathias Homann]

oh, the "use this proxy server for all protocols" does make sense - as long as you keep socks out of it.
here's a "best of both worlds" idea:

• change the code so that socks is not affected by that option
• re-word the text to "Use this proxy for http(s) and ftp", or "Use this proxy for all supported protocols (does not include socks)".

Comment 15

[Kershaw Chang [:kershaw]]

(In reply to Michal Novotny [:michal] from comment #13)

Then the code works as expected. IMO the option "Use this proxy server for all protocols" doesn't make any sense and bug 1577862 made this just obvious.

Sounds like we need to somehow fix this issue. Michal, could you give this bug a priority?

Comment 16

[Michal Novotny [:michal]]

This needs to be fixed in preferences. There is already a bug for this.

Comment 17

[Mathias Homann]

(In reply to Michal Novotny [:michal] from comment #16)

This needs to be fixed in preferences. There is already a bug for this.

...which has not gotten any attention for six years, until now...

Comment 18

[jonathan]

This is causing connection issues for users at the Human Brain Project from certain universities when connecting to Jupyter Notebooks. Is there a suggested workaround?

Comment 19

[Mathias Homann]

there is:

do not use the "Use this proxy for all protocols" setting. Fill in proxy details for each protocol (http, https, ftp, and socks) separately, and make sure you have the socks settings right (look at the bottom half of my screenshot that is attached here).

Comment 20

[Nhi Nguyen (:nhi)]

this depends on bug 969282 getting fixed.

Comment 22

[Cedric]

I am the submitter of bug 1607235 and was looking at this bug after upgrading to 72.0.2 this morning.

The workaround mentioned in this bug - which is supposed to work (do not check 'use for all protocols', and manually entering it for each protocol + setting the right socks version) - does not work at all.

For me it remains the issue as described in bug 1607235: the moment you have it set manually, websockets refuse cooperation.

Steps to reproduce are very easy and can be found in the bug. Would be interesting to see if Mathias can get it to work with his workaround?

Comment 23

[alekiv]

Indeed. Now it is totally broken.

Comment 24

[alekiv]

(In reply to alekiv from comment #23)

Indeed. Now it is totally broken.

False alert. Seems it was some network glitch

Comment 26

[marcel.baltruschat]

Hello, i can reproduce the problem here while trying to access a Moodle instance with integrated Collabora texteditor which is using websockets to connect. I am using a FF 72.0.2 64bit on Windows 10 1909 behind a squid Proxy, what is working for me as a workaround is choosing the Option "Proxy-Einstellungen des System verwenden".

Comment 27

[x]

I confirm the issue in FF 72.0.2. Can´t use Slack or Mattermost etc. It is very annoying.

Comment 28

[Cedric]

(In reply to marcel.baltruschat from comment #26)

Hello, i can reproduce the problem here while trying to access a Moodle instance with integrated Collabora texteditor which is using websockets to connect. I am using a FF 72.0.2 64bit on Windows 10 1909 behind a squid Proxy, what is working for me as a workaround is choosing the Option "Proxy-Einstellungen des System verwenden".

That workaround is another bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1601671 and should probably all get resolved in FF 73.

Comment 29

[Randell Jesup [:jesup] (needinfo me)]

Given that bug 969282 has been fixed (and bug 1601671), is this still an issue?

Comment 30

[BugBot [:suhaib / :marco/ :calixte]]

Redirect needinfos that are pending on inactive users to the triage owner.
:jesup, since the bug has recent activity, could you have a look please?

For more information, please visit BugBot documentation.

Comment 31

[Arrigo Marchiori]

I am still seeing this problem on my Linux systems.
They are Ubuntu 22.04 with Firefox 122.0 (DEB version 122.0+build2-0ubuntu0.22.04.1~mt1)

Environment variables http_proxy, ftp_proxy and https_proxy are set in the form http://myproxy:8888

The network is IPv4 with no default route.

Websocket apps such as https://echo.websocket.org/.ws do not work.

If I add a default route, websocket connections start to work.

Comment 32

[BMO Automation]

Moving bug to Core/Networking: Proxy.

Comment 33

[Kershaw Chang [:kershaw]]

(In reply to Arrigo Marchiori from comment #31)

I am still seeing this problem on my Linux systems.
They are Ubuntu 22.04 with Firefox 122.0 (DEB version 122.0+build2-0ubuntu0.22.04.1~mt1)

Environment variables http_proxy, ftp_proxy and https_proxy are set in the form http://myproxy:8888

The network is IPv4 with no default route.

Websocket apps such as https://echo.websocket.org/.ws do not work.

If I add a default route, websocket connections start to work.

If this is caused by default route, I assume this is not a Firefox issue?
Could you explain more? Thanks.

Comment 34

[Arrigo Marchiori]

(In reply to Kershaw Chang [:kershaw] from comment #33)

(In reply to Arrigo Marchiori from comment #31)
[...]

If I add a default route, websocket connections start to work.

If this is caused by default route, I assume this is not a Firefox issue?
Could you explain more? Thanks.

I am testing two configurations:

1. without a default route;
2. with a default route.

In both configurations:

• Firefox is set to "use system proxy";
• a proxy is indicated with the http_proxy, ftp_proxy and https_proxy environment variables.

I expect websockets to work in both configurations, through the indicated proxy.
Instead, websockets only work in configuration 2.
Looking at network traffic, I see that websockets-related packets are directly exchanged with the target host, instead of being channeled through the proxy. In other words, websockets do not seem to obey the proxy settings from environment variables, but rather only rely on the presence of a default route.

If I set Firefox to "manual proxy configuration" and enter the proxy information directly into Firefox settings, then websockets work in both configurations 1. and 2.
In other words, websockets seem to obey proxy settings only when entered as "manual proxy configuration".

I hope I could explain myself clearly enough.

The above was just verified on Firefox 131.0.2 (64-bit) as currently shipped with Ubuntu 22.04.

Comment 35

[Kershaw Chang [:kershaw]]

Hi, could you try to capture a http log with the steps below?

1. Go to about:logging. Set New log modules to timestamp,sync,nsHttp:5,nsWebSocket:5,nsSocketTransport:5,nsHostResolver:5,proxy:5 and select Logging to a file.
2. Click Start logging.
3. Go to https://echo.websocket.org/.ws
4. Stop logging.
5. Send the log file to [e-mail].

Thanks.

Comment 36

[Arrigo Marchiori]

Hello,

email just sent.

Thanks.

Comment 37

[Valentin Gosu [:valentin] (he/him) {{ AFK in July }}]

Environment variables http_proxy, ftp_proxy and https_proxy are set in the form http://myproxy:8888

That's a good pointer. I looked for where we use these env variables, and got to GetProxyFromEnvironment.
In this case, when called via mozilla::net::nsProtocolProxyService::Resolve_Internal we'd call GetProxyForURI for a ws or wss URI, and check for the "ws_proxy" or "wss_proxy" env variable, which doesn't exist.

I'm thinking we need to change "wss" to "https" and "ws" to "http" before calling mSystemProxySettings->GetProxyForURI ?

Not sure if this is the same as comment 0, but it does seem like this is a valid bug.

Comment 38

[Arrigo Marchiori]

(In reply to Valentin Gosu [:valentin] (he/him) from comment #37)

Environment variables http_proxy, ftp_proxy and https_proxy are set in the form http://myproxy:8888

That's a good pointer. I looked for where we use these env variables, and got to GetProxyFromEnvironment.
In this case, when called via mozilla::net::nsProtocolProxyService::Resolve_Internal we'd call GetProxyForURI for a ws or wss URI, and check for the "ws_proxy" or "wss_proxy" env variable, which doesn't exist.

I'm thinking we need to change "wss" to "https" and "ws" to "http" before calling mSystemProxySettings->GetProxyForURI ?

libcurl (source) checks environment variables ws_proxy and wss_proxy and then, if they are not set, falls back to http_proxy and https_proxy.

I suggest we follow the same logic as libcurl. Others do as well.

Not sure if this is the same as comment 0, but it does seem like this is a valid bug.

:-)

Comment 39

[Valentin Gosu [:valentin] (he/him) {{ AFK in July }}]

Attachment: Bug 1601871 - unix: Fallback to http_proxy or https_proxy for ws and wss loads r=#necko

Comment 40

[Arrigo Marchiori]

Thank you for taking care of this!
Maybe you may want to make the new function GetEnvRetryUppercase() to be static?
(I tried to comment on the Phabricator differential but I was denied "edit" access)

Comment 41

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/344418590372
unix: Fallback to http_proxy or https_proxy for ws and wss loads r=necko-reviewers,kershaw

Comment 42

[pstanciu]

https://hg.mozilla.org/mozilla-central/rev/344418590372

Comment 43

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:valentin, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
• If no, please set status-firefox135 to wontfix.

For more information, please visit BugBot documentation.

Comment 44

[Valentin Gosu [:valentin] (he/him) {{ AFK in July }}]

This has been broken for a very long time.
We can let it ride the trains to avoid any surprises.

Comment 45

[Valentin Gosu [:valentin] (he/him) {{ AFK in July }}]

Could you check the latest Nightly and confirm if this fixes the bug for you?
https://nightly.mozilla.org/