1947229 - Proposal For Proxy Code and UI Enhancement

Status: NEW

(Core :: Networking: Proxy, enhancement, P2)

Description

[erosman [:erosman]]

Proposal For Proxy Code and UI Enhancement

Subsequent to a number discussion in various bugs, please find the following areas that could benefit from enhancement:

• There are inconsistencies with Chrome which complicates cross-browser extension development (which should be easy enough to resolve)
  See also: Inconsistency: proxy API
• The protocol based proxying is rather outdated and no longer used (Bug 1804693)
• Chrome also has protocol based proxying, which can be kept for consistency, or left out (as they are not used anymore)
  e.g. proxyForFtp (dead), proxyForHttp & proxyForHttps
• There are areas where Firefox excels in and these should be maintained (e.g. proxy.onRequest that allows for SOCKS authentication and more)
• Chrome doesn't apply passthrough (no_proxies_on) to PAC URLs while Firefox does and it should be kept
• Chrome's passthrough has extended format (e.g. scheme) comparing to Firefox (not a major issue, but can be considered) (Bug 1810738)
• Proxy DNS option would be removed (to become default) and all DNS queries should be left to the proxy (HTTP & SOCKS) (Bug 1893670, Bug 1741375 -> https://bugzilla.mozilla.org/show_bug.cgi?id=1741375#c30, https://bugzilla.mozilla.org/show_bug.cgi?id=1741375#c33)
• Backward compatibility should easily be maintained by aliasing and minor mapping

Additional areas:

• There are some conflicts with DoH & ESNI which require consideration (Bug 1579985, Secure SNI is disabled if FoxyProxy is enabled)
• WebRTC and DNS leaks when using a proxy (Bug 1791243, Bug 1882276, Bug 1799411)

Update:
Extension related content removed to avoid ambiguity.

Comment 1

[erosman [:erosman]]

I am removing extension related data as it has led to a misunderstanding.
Once necko-proxy is updated, the extension API can reflect the changes accordingly.

Comment 2

[Valentin Gosu [:valentin] (he/him)]

Hi Shane and Luca,

erosman has some proposed changes for out webextension API - see comment 0 and comment 1.
Do you have any input or feedback here? If so, we might want to reflect that on https://github.com/w3c/webextensions/issues/573#issuecomment-2016589524

I'm thinking this bug might belong in the webextension component, to work out all of the kinks in the API, but probably the necko work referenced in the other bugs needs to happen in step with the API changes.
Let me know what you think. Thanks!

Comment 3

[erosman [:erosman]]

Please note that the proposed changes are for the Firefox's proxy process (not the WebExtension API).

In other words...

// keep
"network.proxy.type",
"network.proxy.no_proxies_on",
"network.proxy.autoconfig_url",
"signon.autologin.proxy",
"network.http.proxy.respect-be-conservative",

// add
"network.proxy.single_proxy",
  "network.proxy.single_proxy.scheme",
  "network.proxy.single_proxy.host",
  "network.proxy.single_proxy.port",

// remove/deprecate
"network.proxy.http",
"network.proxy.http_port",
"network.proxy.share_proxy_settings",
"network.proxy.ssl",
"network.proxy.ssl_port",
"network.proxy.socks",
"network.proxy.socks_port",
"network.proxy.socks_version",
"network.proxy.socks_remote_dns",
"network.proxy.socks5_remote_dns",

Comment 4

[erosman [:erosman]]

Attachment: popup.png

Here is a quick example of how "Connection Settings" could look.

Comment 5

[erosman [:erosman]]

Attachment: popup2.png

Only for better visual representation, here is an updated screenshot of how "Connection Settings" could look.

Comment 6

[erosman [:erosman]]

Attachment: popup3.png

Updated Image

Comment 7

[erosman [:erosman]]

Attachment: popup4.png

Displaying the scheme select

Comment 8

[Luca Greco [:rpl] [:luca] [:lgreco]]

(In reply to Valentin Gosu [:valentin] (he/him) from comment #2)

Hi Shane and Luca,

erosman has some proposed changes for out webextension API - see comment 0 and comment 1.
Do you have any input or feedback here? If so, we might want to reflect that on https://github.com/w3c/webextensions/issues/573#issuecomment-2016589524

I'm thinking this bug might belong in the webextension component, to work out all of the kinks in the API, but probably the necko work referenced in the other bugs needs to happen in step with the API changes.
Let me know what you think. Thanks!

Comment 3 seems to suggets this bug would be meant to be tracking necko side of the underlying proxying implementation (plus some points related to the proxy-related about:preferences UI?), nonetheless comment 0 and the linked github issue seems to also includes things that feels like more on the webextensions API side, and I wouldn't exclude that even changes on the necko side would also require changes on the proxy API implementation.

I'm redirecting this needinfo to Rob, through his regular participation to the WECG meetings he may have additional context around the cross-browser inconsistency described in the github issue, and he may also know of other existing bugzilla issues tracking those inconsistency (or a subset of them).

Comment 9

[erosman [:erosman]]

Please allow me to simplify the topic.

necko-proxy

1. Firefox (unlike Chrome) currently lacks a viable method to set a single proxy for all connections. The requirement is to be able to set a single host, port to any scheme.
   Following is an example of Firefox pref that could be used. Attached images show how the new (simplified) UI could look.

"network.proxy.single_proxy.scheme",        // "http", "https", "socks4", "socks5", "quic"
"network.proxy.single_proxy.host",
"network.proxy.single_proxy.port",

2. While working on the necko-proxy's underlying code, there are a number of proxy-related bug reports that would make sense to address.
3. Currently Firefox has other pref associated with proxy, some of which can be deprecated.
4. For backward compatibility, I would suggest mapping some of the old prefs to the new pref/process.

WebExtension API

Once necko-proxy is updated, the necessary few changes to the proxy.settings API can be made accordingly, to reflect the changes to the necko-proxy.
Another bug will be filed for this purpose.

Comment 10

[Valentin Gosu [:valentin] (he/him)]

Thanks, that's a good proposal.
Not sure about the single_proxy name, but I'm sure we can find a more insightful name for it.
We should also try to land the UI change in this bug.
I think the most difficult part of this bug is migrating the old prefs (or making sure we don't override their settings).

Comment 11

[erosman [:erosman]]

Not sure about the single_proxy name, but I'm sure we can find a more insightful name for it.

That was only an example. The reason I used it was since chrome.proxy: ProxyRules uses singleProxy and I thought it could help with cross-browser development.

I think the most difficult part of this bug is migrating the old prefs (or making sure we don't override their settings).

Here are some points to consider:

1. PAC processing must remain as it is now (with synchronous DNS)
   (it might be worthwhile to have its process separated, if not already separated)

2. Using SOCKS4 & SOCKS5 to eliminate ambiguity (same as chromium)
   There are sections where SOCKS refers to SOCK4 while in others it refers to SOCKS5

3. Generating some statists on the actual use of these pref
   We would need to know if there are still people using protocol based proxying.
   The data needed is how many users have set all the followings together (not blank) and to "different" values.

"network.proxy.http",
"network.proxy.http_port",
"network.proxy.ssl",
"network.proxy.ssl_port",

4. Based on the above statistics, old prefs can be mapped to the new prefs.

Using single_proxy as an example

"network.proxy.http",                       -> "network.proxy.single_proxy.host",
"network.proxy.http_port",                  -> "network.proxy.single_proxy.port",
                                            -> "network.proxy.single_proxy.scheme": "http"

"network.proxy.share_proxy_settings",       -> deprecate (will be default)

"network.proxy.ssl",                        -> "network.proxy.single_proxy.host",
"network.proxy.ssl_port",                   -> "network.proxy.single_proxy.port",
                                            -> "network.proxy.single_proxy.scheme": "https"

"network.proxy.socks",                      -> "network.proxy.single_proxy.host",
"network.proxy.socks_port",                 -> "network.proxy.single_proxy.port",
"network.proxy.socks_version",              -> "network.proxy.single_proxy.scheme": "socks4" | "socks5"

"network.proxy.socks_remote_dns",           -> deprecate (will be default)
"network.proxy.socks5_remote_dns",          -> deprecate (will be default)

Comment 12

[Valentin Gosu [:valentin] (he/him)]

Seems like a good plan.

There are sections where SOCKS refers to SOCK4 while in others it refers to SOCKS5

Do you have an example of that?

The data needed is how many users have set all the followings together (not blank) and to "different" values.

This part will be a bit difficult to get right.

Here's some of the proxy telemetry we do currently have.
https://glam.telemetry.mozilla.org/firefox/probe/http_proxy_type/
https://glam.telemetry.mozilla.org/fog/probe/http_proxy_type/
https://glam.telemetry.mozilla.org/fog/probe/networking_proxy_info_type/
https://glam.telemetry.mozilla.org/fog/probe/network_proxy_settings_proxy_type_preference/

As far as I recall, the proxy settings are some of the most accessed settings in Firefox, so we'll have to treat this with care. 🙂

Comment 13

[erosman [:erosman]]

Do you have an example of that?

In the early days, there was only SOCKS. Later when SOCKS5 appeared, the PAC adopted SOCKS (meaning 4) and SOCKS5.
Later to clarify, SOCKS4 keyword was added but PAC standard remained as it was.

socks means socks5
https://searchfox.org/mozilla-central/source/toolkit/components/extensions/ProxyChannelFilter.sys.mjs#48

uses socks/socks4/socks5
https://searchfox.org/mozilla-central/source/netwerk/base/ProxyConfig.h#122

socks means socks4
https://searchfox.org/mozilla-central/source/browser/components/enterprisepolicies/helpers/ProxyPolicies.sys.mjs#35

Here is a comment that I had prepared when I was deciding on the keyword to use in the extension. I decided on using SOCK4/SOCK5 to remove any doubt. (see also Bug 1378205)

----- SOCKS keyword -----
https://developer.chrome.com/docs/extensions/reference/proxy/#proxy-rules
https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/proxy/ProxyInfo#type
https://chromium.googlesource.com/chromium/src/+/HEAD/net/docs/proxy.md#http-proxy-scheme
Chrome PAC schemes  PROXY | HTTPS | SOCKS4/SOCKS | SOCKS5
Chrome API schemes  http | https | socks4 | socks5 | quic

Firefox PAC schemes PROXY/HTTP | HTTPS | SOCKS4/SOCKS | SOCKS5
Firefox API types   http | https | socks4 | socks (means socks5) | direct

Firefox/Chrome PAC 'SOCKS' means SOCKS4
Firefox API 'SOCKS' means SOCKS5

Here's some of the proxy telemetry we do currently have.

It doesn’t show them to me. It may require login!?

The point is, only if a user has set one proxy for HTTP and a different one for HTTPS, there could be a complication.
TBH, I would be surprised if anyone has done it this way.
Proxy servers have been supporting both HTTP and HTTPS at the same time for decades.
Separate proxy for HTTP and HTTPS was only relevant in the early days of HTTPS.
As a proxy user for 25+ years and a proxy extension developer, I would never use protocol based proxying.

footnote
If absolute assurance is desired, code can keep supporting http only and https only (although it might be a waste).
If so, alignment with Chrome can be considered through proxyForHttp & proxyForHttps APIs.
They are the same as the single_proxy example but only forward based on protocol (http -> http & https -> https).