Closed Bug 1601871 Opened 6 years ago Closed 6 months ago

REGRESSION: Websocket connection behind HTTP proxy does not work

Categories

(Core :: Networking: Proxy, defect, P2)

71 Branch
defect
Points:
2

Tracking

()

RESOLVED FIXED
136 Branch
Tracking Status
firefox-esr115 --- wontfix
firefox-esr128 --- wontfix
firefox134 --- wontfix
firefox135 --- wontfix
firefox136 --- fixed

People

(Reporter: Mathias.Homann, Assigned: valentin, NeedInfo)

References

(Regression)

Details

(Keywords: regression, Whiteboard: [necko-triaged][necko-priority-next])

Attachments

(2 files)

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:71.0) Gecko/20100101 Firefox/71.0

Steps to reproduce:

I'm trying to use a website that uses websckets from behind a HTTP proxy

Steps to reproduce:

Actual results:

"it does not work"

Expected results:

It should work, it used to in FF 70 and before.

Possibly a regression of https://bugzilla.mozilla.org/show_bug.cgi?id=1177909 and/or https://bugzilla.mozilla.org/show_bug.cgi?id=1517782

additional info:

just having a http proxy configured and in use breaks it. I have put the html test page from https://www.websocket.org/echo.html on my local server, which is bypassed from going through the proxy, and that already breaks it. As soon as I turn the proxy OFF, it works.

Component: Untriaged → Networking: WebSockets
Product: Firefox → Core

another test: even if the websockets test page is on localhost and localhost is explicitely excluded in the proxy settings it is broken.

Same there. After upgrade from 70.0.1 to 70.1 Websockets connections through proxy don't work.

Can you please try to run
https://mozilla.github.io/mozregression/

Thank you!

Flags: needinfo?(Mathias.Homann)

sorry, can't, at least on linux, too many python version conflicts.

isn't there a python3 version? python 2 is end of life!

I'll try on windows.

ok so I've gone through the process with mozregression-gui on windows, now what?

here's the end of the output:

2019-12-07T18:08:52: INFO : Narrowed inbound regression window from [caa26030, bb918aa3] (3 builds) to [a417315a, bb918aa3] (2 builds) (~1 steps left)
2019-12-07T18:08:52: DEBUG : Starting merge handling...
2019-12-07T18:08:52: DEBUG : Using url: https://hg.mozilla.org/integration/autoland/json-pushes?changeset=bb918aa3c389d3e78e5cbefffbacdb0faefea6a0&full=1
2019-12-07T18:08:53: DEBUG : Found commit message:
Bug 1562664 - Replace XUL textbox usages with HTML inputs in editBookmarkPanel.inc.xul r=bgrins

Differential Revision: https://phabricator.services.mozilla.com/D36495

2019-12-07T18:08:53: DEBUG : Did not find a branch, checking all integration branches
2019-12-07T18:08:53: INFO : The bisection is done.
2019-12-07T18:08:53: INFO : Stopped

the full log file is here:
https://gist.github.com/lemmy04/ba9629dd4ac1ea19cff8cfc8d3daf775

Flags: needinfo?(Mathias.Homann)

from comment#6 full log
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=15ffd69c&tochange=8b8ce461

I suspect the culprit bug is a417315a75d54d0c89dba01b5fea9a9fe6e92ee8 Michal Novotny — Bug 1577862 - Websockets should prioritize SOCKS5 proxies over HTTP(S) proxies, r=valentin

Has Regression Range: --- → yes
Keywords: regression
Regressed by: 1577862

Here's an interesting little detail which kinda goes with that suspicion:

when I turn on "use this proxy for everything" it tries to use the squid proxy on port 3128 for socks5 as well. and fails.
when I don't turn that on, and instead put the proper port in for the socks setup (1080), websockets work fine even with the proxy use enabled.

food for thought:

  • it might be said that this is actually a "works as designed" case
  • in that case, should "use this proxy for all protocols" really include socks as well? I don't know if there are any proxies that do support socks AS WELL as all the rest, but I do know that squid doesn't...

(In reply to Mathias Homann from comment #8)

when I don't turn that on, and instead put the proper port in for the socks setup (1080), websockets work fine even with the proxy use enabled.

What did you have set as host:port for SOCKS when it didn't work? Bug 1577862 changed preferred proxy type for websockets. If you have no host and port specified for SOCKS proxy, HTTP proxy should be used.

Flags: needinfo?(Mathias.Homann)

that is my point: when you select the option "use this proxy for all protocols" it puts that proxy and port into the socks settings as well, which is plain wrong - it should not overwrite what is already there unless the field was left empty.
I don't know if there even are HTTP proxy services that support socks on the same port.

See attached image for clarification.

Flags: needinfo?(Mathias.Homann)

Then the code works as expected. IMO the option "Use this proxy server for all protocols" doesn't make any sense and bug 1577862 made this just obvious.

oh, the "use this proxy server for all protocols" does make sense - as long as you keep socks out of it.
here's a "best of both worlds" idea:

  • change the code so that socks is not affected by that option
  • re-word the text to "Use this proxy for http(s) and ftp", or "Use this proxy for all supported protocols (does not include socks)".

(In reply to Michal Novotny [:michal] from comment #13)

Then the code works as expected. IMO the option "Use this proxy server for all protocols" doesn't make any sense and bug 1577862 made this just obvious.

Sounds like we need to somehow fix this issue. Michal, could you give this bug a priority?

Flags: needinfo?(michal.novotny)
Priority: -- → P3
Whiteboard: [necko-triaged]
Priority: P3 → --

This needs to be fixed in preferences. There is already a bug for this.

Depends on: 969282
Flags: needinfo?(michal.novotny)
Priority: -- → P3

(In reply to Michal Novotny [:michal] from comment #16)

This needs to be fixed in preferences. There is already a bug for this.

...which has not gotten any attention for six years, until now...

This is causing connection issues for users at the Human Brain Project from certain universities when connecting to Jupyter Notebooks. Is there a suggested workaround?

there is:

do not use the "Use this proxy for all protocols" setting. Fill in proxy details for each protocol (http, https, ftp, and socks) separately, and make sure you have the socks settings right (look at the bottom half of my screenshot that is attached here).

Flags: needinfo?(nhnguyen)

this depends on bug 969282 getting fixed.

Flags: needinfo?(nhnguyen)
See Also: → 1608090

I am the submitter of bug 1607235 and was looking at this bug after upgrading to 72.0.2 this morning.

The workaround mentioned in this bug - which is supposed to work (do not check 'use for all protocols', and manually entering it for each protocol + setting the right socks version) - does not work at all.

For me it remains the issue as described in bug 1607235: the moment you have it set manually, websockets refuse cooperation.

Steps to reproduce are very easy and can be found in the bug. Would be interesting to see if Mathias can get it to work with his workaround?

Indeed. Now it is totally broken.

(In reply to alekiv from comment #23)

Indeed. Now it is totally broken.

False alert. Seems it was some network glitch

Hello, i can reproduce the problem here while trying to access a Moodle instance with integrated Collabora texteditor which is using websockets to connect. I am using a FF 72.0.2 64bit on Windows 10 1909 behind a squid Proxy, what is working for me as a workaround is choosing the Option "Proxy-Einstellungen des System verwenden".

I confirm the issue in FF 72.0.2. Can´t use Slack or Mattermost etc. It is very annoying.

(In reply to marcel.baltruschat from comment #26)

Hello, i can reproduce the problem here while trying to access a Moodle instance with integrated Collabora texteditor which is using websockets to connect. I am using a FF 72.0.2 64bit on Windows 10 1909 behind a squid Proxy, what is working for me as a workaround is choosing the Option "Proxy-Einstellungen des System verwenden".

That workaround is another bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1601671 and should probably all get resolved in FF 73.

Severity: normal → S3

Given that bug 969282 has been fixed (and bug 1601671), is this still an issue?

Flags: needinfo?(marcel.baltruschat)
Flags: needinfo?(cibet14545)
Flags: needinfo?(cedric.vanlabeke)
Whiteboard: [necko-triaged] → [necko-triaged][necko-monitor]

Redirect needinfos that are pending on inactive users to the triage owner.
:jesup, since the bug has recent activity, could you have a look please?

For more information, please visit BugBot documentation.

Flags: needinfo?(rjesup)
Flags: needinfo?(marcel.baltruschat)
Flags: needinfo?(cibet14545)
Flags: needinfo?(cedric.vanlabeke)

I am still seeing this problem on my Linux systems.
They are Ubuntu 22.04 with Firefox 122.0 (DEB version 122.0+build2-0ubuntu0.22.04.1~mt1)

Environment variables http_proxy, ftp_proxy and https_proxy are set in the form http://myproxy:8888

The network is IPv4 with no default route.

Websocket apps such as https://echo.websocket.org/.ws do not work.

If I add a default route, websocket connections start to work.

Moving bug to Core/Networking: Proxy.

Component: Networking: WebSockets → Networking: Proxy
Flags: needinfo?(rjesup)
Priority: P3 → P2
Whiteboard: [necko-triaged][necko-monitor] → [necko-triaged][necko-new]
Whiteboard: [necko-triaged][necko-new] → [necko-triaged][necko-priority-new]
Points: --- → 5
Rank: 3
Whiteboard: [necko-triaged][necko-priority-new] → [necko-triaged][necko-priority-next]

(In reply to Arrigo Marchiori from comment #31)

I am still seeing this problem on my Linux systems.
They are Ubuntu 22.04 with Firefox 122.0 (DEB version 122.0+build2-0ubuntu0.22.04.1~mt1)

Environment variables http_proxy, ftp_proxy and https_proxy are set in the form http://myproxy:8888

The network is IPv4 with no default route.

Websocket apps such as https://echo.websocket.org/.ws do not work.

If I add a default route, websocket connections start to work.

If this is caused by default route, I assume this is not a Firefox issue?
Could you explain more? Thanks.

Flags: needinfo?(ardovm)

(In reply to Kershaw Chang [:kershaw] from comment #33)

(In reply to Arrigo Marchiori from comment #31)
[...]

If I add a default route, websocket connections start to work.

If this is caused by default route, I assume this is not a Firefox issue?
Could you explain more? Thanks.

I am testing two configurations:

  1. without a default route;
  2. with a default route.

In both configurations:

  • Firefox is set to "use system proxy";
  • a proxy is indicated with the http_proxy, ftp_proxy and https_proxy environment variables.

I expect websockets to work in both configurations, through the indicated proxy.
Instead, websockets only work in configuration 2.
Looking at network traffic, I see that websockets-related packets are directly exchanged with the target host, instead of being channeled through the proxy. In other words, websockets do not seem to obey the proxy settings from environment variables, but rather only rely on the presence of a default route.

If I set Firefox to "manual proxy configuration" and enter the proxy information directly into Firefox settings, then websockets work in both configurations 1. and 2.
In other words, websockets seem to obey proxy settings only when entered as "manual proxy configuration".

I hope I could explain myself clearly enough.

The above was just verified on Firefox 131.0.2 (64-bit) as currently shipped with Ubuntu 22.04.

Flags: needinfo?(ardovm)
Flags: needinfo?(kershaw)

Hi, could you try to capture a http log with the steps below?

  1. Go to about:logging. Set New log modules to timestamp,sync,nsHttp:5,nsWebSocket:5,nsSocketTransport:5,nsHostResolver:5,proxy:5 and select Logging to a file.
  2. Click Start logging.
  3. Go to https://echo.websocket.org/.ws
  4. Stop logging.
  5. Send the log file to necko@mozilla.com.

Thanks.

Flags: needinfo?(kershaw) → needinfo?(ardovm)

Hello,

email just sent.

Thanks.

Flags: needinfo?(ardovm)

Environment variables http_proxy, ftp_proxy and https_proxy are set in the form http://myproxy:8888

That's a good pointer. I looked for where we use these env variables, and got to GetProxyFromEnvironment.
In this case, when called via mozilla::net::nsProtocolProxyService::Resolve_Internal we'd call GetProxyForURI for a ws or wss URI, and check for the "ws_proxy" or "wss_proxy" env variable, which doesn't exist.

I'm thinking we need to change "wss" to "https" and "ws" to "http" before calling mSystemProxySettings->GetProxyForURI ?

Not sure if this is the same as comment 0, but it does seem like this is a valid bug.

Status: UNCONFIRMED → NEW
Points: 5 → 2
Ever confirmed: true

(In reply to Valentin Gosu [:valentin] (he/him) from comment #37)

Environment variables http_proxy, ftp_proxy and https_proxy are set in the form http://myproxy:8888

That's a good pointer. I looked for where we use these env variables, and got to GetProxyFromEnvironment.
In this case, when called via mozilla::net::nsProtocolProxyService::Resolve_Internal we'd call GetProxyForURI for a ws or wss URI, and check for the "ws_proxy" or "wss_proxy" env variable, which doesn't exist.

I'm thinking we need to change "wss" to "https" and "ws" to "http" before calling mSystemProxySettings->GetProxyForURI ?

libcurl (source) checks environment variables ws_proxy and wss_proxy and then, if they are not set, falls back to http_proxy and https_proxy.

I suggest we follow the same logic as libcurl. Others do as well.

Not sure if this is the same as comment 0, but it does seem like this is a valid bug.

:-)

Assignee: nobody → valentin.gosu
Status: NEW → ASSIGNED

Thank you for taking care of this!
Maybe you may want to make the new function GetEnvRetryUppercase() to be static?
(I tried to comment on the Phabricator differential but I was denied "edit" access)

Pushed by valentin.gosu@gmail.com: https://hg.mozilla.org/integration/autoland/rev/344418590372 unix: Fallback to http_proxy or https_proxy for ws and wss loads r=necko-reviewers,kershaw
Status: ASSIGNED → RESOLVED
Closed: 6 months ago
Resolution: --- → FIXED
Target Milestone: --- → 136 Branch

The patch landed in nightly and beta is affected.
:valentin, is this bug important enough to require an uplift?

  • If yes, please nominate the patch for beta approval.
  • If no, please set status-firefox135 to wontfix.

For more information, please visit BugBot documentation.

Flags: needinfo?(valentin.gosu)

This has been broken for a very long time.
We can let it ride the trains to avoid any surprises.

Flags: needinfo?(valentin.gosu)

Could you check the latest Nightly and confirm if this fixes the bug for you?
https://nightly.mozilla.org/

Flags: needinfo?(ardovm)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: