1412790 - Alternative Approach to WebExtension PAC Implementation

Status: RESOLVED DUPLICATE of bug 1425535

(WebExtensions :: Request Handling, enhancement, P5)

Description

[erosman [:erosman]]

User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0
Build ID: 20170921100141

Steps to reproduce:

At the moment, the Proxy API requires further development to implement the missing PAC functions and more. (bug 1353510)

I was thinking about a totally different approach AFA PAC implementation is concerned.

Currently Firefox already has everything needed but it is not accessible from WebExtensions.

Instead of creating (duplicating) the aforementioned functions it would be feasible to give a very limited access to the preferences (similar to "chrome_settings_overrides" or "chrome_url_overrides") and let Firefox handle the rest.

Access is ONLY needed for the following preferences: (e.g. "chrome_proxy_overrides" :p )

'network.proxy.autoconfig_url'
'network.proxy.type'
'signon.autologin.proxy'

The only additional consideration would be the location of the PAC (i.e. URL)

Firefox does have access to local filesystem so in theory there shouldn't be any issue in setting a local file URL or a remote URLs.

Alternately, it might also be useful to be able to pass the entire PAC as DATA:URI to 'network.proxy.autoconfig_url'.

Above settings should not pose any extra risk and they are readily available to users via Firefox Options.

Comment 1

[Caitlin Neiman [:caitmuenster]]

Hi erosman, this has been added to the agenda for the November 14 WebExtensions APIs triage meeting. Would you be able to join us? 

Here's a quick overview of what to expect at the triage: 

* We normally spend 5 minutes per bug
* The more information in the bug, the better
* The goal of the triage is to give a general thumbs up or thumbs down on a proposal; we won't be going deep into implementation details

Relevant Links: 

* Wiki for the meeting: https://wiki.mozilla.org/Add-ons/Contribute/Triage
* Meeting agenda: https://docs.google.com/document/d/1g3RMfKZ3671NcusMqkoOiKwfPekRe-VI7Rzqxo6F_Ao/edit#
* Vision doc for WebExtensions: https://wiki.mozilla.org/WebExtensions/Vision

Comment 2

[Eric Jung [:ericjung]]

This implementation contrasts with bug 1396485 (and related: bug 1319630, bug 1319631). We should do this bug or bug 1396485 but not both. See also https://docs.google.com/document/d/17DqDXWc3RMWH_LPUkjZcSmtqXHi1mpz6T-ocSey41Pg/edit where Shane and I discuss how to handle PAC on page 3 using the existing, preference-less design.

Comment 3

[Caitlin Neiman [:caitmuenster]]

Shane, we wanted to get your feedback before approving or rejecting this proposal.

Comment 4

[Shane Caraveo (:mixedpuppy)]

I'd actually already been thinking about this separately and hadn't seen this bug.  While handling the prefs would be easy, we then also need to add UI in prefs to handle the choices of extensions making pref modifications.  We've done this for other prefs, so I don't think that's a big deal, but lets ask Mark... 

(also cc Bob since he's done lots of this kind of stuff)

Regarding Eric's comment #2, if this solves problems that resulted in some of those bugs, I'd rather go this approach.  I don't think the conflicts with the goals outlined in the google doc, since that has to happen anyway to get away from synchronous pac scripts in extensions.  I'm really not interested in the new return values.

Leaving ni on myself till we get input on above comments.

Comment 5

[Mark Striemer [:mstriemer]]

The proxy settings look a bit different than the other settings we've done this for. Proxy settings are in a modal instead in the page directly but I think we can work around this.

To be clear: PAC is Proxy AutoConfig? So essentially an extension wants to set the type in the modal to auto config and define that URL? Then the do not prompt checkbox, as well.

I'd need to run it past Markus but I have a couple ideas of how to do this for the UI. It shouldn't be a concern, I'd say.

Comment 6

[erosman [:erosman]]

That is correct, PAC is Proxy Auto-Config.

A user can go to Firefox Options -> Connection Settings -> and set these values (they are not hidden/internal prefs)

Firefox Options:
- Configure Proxies to Access the Internet (radio) (pref: 'network.proxy.type') needed to select "Automatic proxy configuration URL"

- PAC URL input text (pref: 'network.proxy.autoconfig_url')

- Do not prompt for authentication if password is saved (pref: 'signon.autologin.proxy'), handy to prevent regular auth popup requests.


Important Note: In order to activate a PAC, Firefox requires that the PAC is reloaded (as per RELOAD button). In legacy extension, it was managed with
> Components.classes['@mozilla.org/network/protocol-proxy-service;1'].getService().reloadPAC();

Notes on PAC URL:

- Firefox accepts remote URLs (good)
- Firefox accepts local URLS (e.g. file:///..... good)
- I dont know if Firefox accepts Data:URI as PAC URL !! It may not be imperative but it would be greatly helpful in order to avoid reading local filesystem.

In order to have a dynamic PAC (for an example look at my FoxyPAC extension), PAC needs to be easily editable. PAC as DATA:URI facilitates it greatly.

Otherwise, using a local file system means manually reading the file (file input + FileReader), editing PAC in extension, saving it to local file (download API), setting the FF Option, reloading the PAC (with local file URL).

Comment 7

[Eric Jung [:ericjung]]

Realize what you are doing if this bug lands: nsIProtocolProxyFilter.applyFilter() and network.proxy.* prefs do not work together right now. It's one or the other.

So how is an addon going to support both external PAC files and internal proxy switching by URL? How is the proxyAPI going to support that?

erosman is only thinking of addons that support external PAC files (his FoxyPAC addon). He's not thinking of addons that want to support both external PAC files and internal proxy switching by URL (FoxyProxy, Proxy SwitchyOmega, others)

I urge you to reconsider this implementation, or at least explain how the above is going to be handled if this bug lands.

Comment 8

[erosman [:erosman]]

> So how is an addon going to support both external PAC files and internal proxy switching by URL? How is the proxyAPI going to support that?

Firefox core supports both PAC and proxy. My suggestion was to pass both back to Firefox and let Firefox handle it, the same way that the legacy addons worked. 

The legacy API only provided access to Firefox internal processes and did not create new API/processes. The WebExtension could also adopt the same approach by providing access, instead of creating new processes.


> erosman is only thinking of addons that support external PAC files (his FoxyPAC addon). He's not thinking of addons that want to support both external PAC files and internal proxy switching by URL (FoxyProxy, Proxy SwitchyOmega, others)

Not true. My suggestion would work for external PAC, internal PAC, local PAC and any other types of PAC. It would work for static PAC and dynamic PAC.
I can rewrite FoxyProxy completely with all its functions, with only using the above 3 preferences. Please dont think that I only have my own interest at heart. ;)
The same applies to any other Proxy addon that is using the current WE Proxy API.

Besides the simplification of the process, it would also provide access to the missing PAC functions that will not be possible in WE Proxy API. In fact, a PAC implementation without the PAC functions is rather limited.

The "internal proxy switching by URL" can also be handled by PAC (Internal) without the need for additional API. I am not thinking about my own add-on (with only 38 users). I am thinking of a way to go forward as the current PAC implementation of WE Proxy API is far too limited.

The bug topic itself relates to PAC implementation only.

Comment 9

[Pedro Côrte-Real]

I'm the author of the ProxySwitcheroo extension[1] and support this proposal. The point of my extension is to have a very simple way to enable/disable proxying. For this I've had to implement a PAC and replicate part of the proxy settings UI. If I wanted to make it support all proxying options I'd have to replicate a bunch of extra UI and request extra (and scary) permissions like access to all browsing data (for HTTP/HTTPS auth). With this proposal the extension would become trivial to implement (just flip "network.proxy.type") and lead to a much better user experience as almost all the config is in the normal proxy preferences (I'd just need a switch to decide which option to switch to/from in the proxy settings).

[1] https://addons.mozilla.org/en-US/firefox/addon/proxyswitcheroo/

Comment 10

[Shane Caraveo (:mixedpuppy)]

(In reply to Eric Jung [:ericjung] from comment #7)

> So how is an addon going to support both external PAC files and internal
> proxy switching by URL?

How did you do it before?  And how does exposing these change that, considering legacy addons had access to this already?

I'm tending to agree with comment 8.  Exposing more of the internal proxy handling is no different than what was available to legacy extensions.  Any proxy extension would be able to interrogate the settings and know what will/wont work.  Right now barring another alternative, I see this as the best way to provide some support for remote configurations in a shorter time frame.  The existing api is unlikely to ever support remote pac scripts.

I'm not entirely convinced of allowing data URIs if the proxy service doesn't already support it.  OTOH it's likely no greater security concern that pointing to an external resource.

We still have to address the proxy filter capability (Bug 1152332) so we can get rid of the current/separate psuedo-pac proxy api and simply have a filter function in the background script.

Comment 11

[erosman [:erosman]]

> I'm not entirely convinced of allowing data URIs if the proxy service doesn't already support it.  OTOH it's likely no greater security concern that pointing to an external resource.

As you have mentioned, passing PAC as data should not add any additional risk in comparison to a remote resource which is read and evaluated the same way DAT:Uri would be. In fact, remotely located PAC are always a concern in addon reviews as we don't know where users are being redirected to.

In order to have a dynamic PAC (where patterns can be changed easily according to user prefs), a system for regularly changing PAC content is needed.

- Dynamic PAC is not possible with Remote PAC which only suit static PAC (good for pass/proxy all option where all are passed through proxy). (N.B. above note)

- Dynamic PAC was possible with local PAC file when extensions could read/write to local filesystem which is not readily available in WE API but it might be possible to manage with some creative coding. However, it would require multiple user initiated steps which is not that user-friendly (as explained in comment #6).


- Any method of passing code directly to 'network.proxy.autoconfig_url' e.g Data:Uri, Bolb etc would greatly improve versatility of the process and allow dynamic PAC to be created with ease.

Comment 12

[Eric Jung [:ericjung]]

> How did you do it before?

Internal proxy switching by URL was done with nsIProtocolProxyFilter.applyFilter(). Remote PAC files were supported by loading the PAC file with XMLHttpRequest, creating a sandbox and importing the PAC functions into it. Then call Components.utils.evalInSandbox(). This is the same way firefox handles PACs internally. It's not a lot of code.

> And how does exposing these change that, considering legacy addons had access to this already?

Legacy addons had access to network.proxy.* prefs, but I am not aware of any that used them with nsIProtocolProxyFilter.applyFilter(). Its one or the other. Having said that, if you go down the path of using network.proxy.* prefs for remote PAC, WPAD, and System Settings support, then something (the addon or the W.E. API) will have to call proxy.unregister() when the prefs are in use. Otherwise, last time I tested this, the prefs won't have any effect. And that is the core of the issue I'm trying to point out: an addon won't be able to support multiple proxy configurations with one using remote PACs and another using, say, an HTTP proxy on 192.168.1.1:3128.

Example:

User is on a censored network (e.g. school, employer) that uses a product like Cisco Web Security Appliance, Trend Micro IWSVA, Symantec Secure Web Gateway, or one of many others in order to censor Facebook, Twitter, YouTube, etc. They do this with a PAC file that is hosted on an intranet address; e.g. 192.168.1.*.

Imagine this user wants to access Facebook, Twitter, YouTube, etc. So he wants to load only blocked sites through an HTTP proxy server at 185.9.208.67:3128. All other URLs should be resolved by the remote PAC file at 192.168.1.*. In fact, any intranet-only sites, for example something like http://portal.home *must* be loaded locally (and not through the HTTP proxy server) because they are only accessible on the local intranet.

How can we support this common scenario with this ticket?

Comment 13

[erosman [:erosman]]

> Imagine this user wants to access Facebook, Twitter, YouTube, etc. So he wants to load only blocked sites through an HTTP proxy server at 185.9.208.67:3128. All other URLs should be resolved by the remote PAC file at 192.168.1.*. 

I did the similar in my FoxyPAC only using local PAC.
This one has preferences attached to it, but you get the idea ;)


function FindProxyForURL(url, host) {

  switch (true) {

    // --- Private Networks
    case private10 && isInNet(dnsResolve(host), '10.0.0.0', '255.0.0.0'):
    case private172 && isInNet(host, '172.16.0.0',  '255.240.0.0'):
    case private192 && isInNet(host, '192.168.0.0',  '255.255.0.0'):
    case localhost && isInNet(host, '127.0.0.0', '255.255.255.0'):
    case plainhost && isPlainHostName(host):
      return 'DIRECT';

    default:
      for (let i = 0, len = patterns.length; i < len; i++) {
        if (patterns[i].test(url)) { return proxies[i] || proxies[0]; }
      }
      return 'DIRECT';
  }
}

There are (were!?) some limitation in the way Firefox handles PAC which are to do with the number of connections going through PAC but in general operation, I haven't come across a situation that couldn't be handled with the PAC (even though using PAC is an old method).

Comment 14

[Eric Jung [:ericjung]]

> I did the similar in my FoxyPAC only using local PAC.

Your answer shows how to use a local PAC to route based on URL patterns and some local IP addresses. Never does it call/invoke/defer to a remote PAC *in addition to switching by URL patterns*.

Comment 15

[erosman [:erosman]]

> Your answer shows how to use a local PAC to route based on URL patterns and some local IP addresses. Never does it call/invoke/defer to a remote PAC *in addition to switching by URL patterns*.

A remote PAC, by definition, can not be dynamically adjustable by the user preferences. That is to say, to allow user to choose the domains that users want to pass through proxy. Therefore, the question of Remote PAC for this purpose is irrelevant.

There is no reason for a user to want to pass a set of domains (local or remote) through a proxy set by the user, and some through settings set by someone else in a remote PAC. All can be handled by one PAC.

In my own FoxyTab, I have provided the function to allow user to spread domains through multiple proxies which means
Domains A, B, C, through proxy A
Domains D, E, F, through proxy B
Domain G through proxy C
etc

A similar approach can be implemented in FoxyProxy as well, if spreading through multiple proxies is needed.


In fact, remote PACs are not allowed on AMO since there is no control over where the client is being redirected to.

If a user personally wishes to use a remote PAC, the user can always set it via going to the Firefox options.

In FoxyProxy (or similar) the function to allow users to choose to pass domain A through proxy and not domain B, can be handled in PAC and PAC in turn can be handled by the proposal.

Comment 16

[Eric Jung [:ericjung]]

> There is no reason for a user to want to pass a set of domains (local or remote) through a proxy set by the user,
> and some through settings set by someone else in a remote PAC. All can be handled by one PAC.

But I gave an example. University, school, and corporate environments. They use remote PACs and, in fact, sometimes block all web traffic that doesn't use it (that is why it is typical to use an ssh -D SOCKS5 server to bypass--it opens an ssh tunnel from client to a server outside the restricted network and a SOCKS5 server on top of it).

> Therefore, the question of Remote PAC for this purpose is irrelevant.

It is relevant in the sense that people want to switch between using one and not using one for specific URLs. There are legacy addons that do precisely this.

> In my own FoxyTab, I have provided the function to allow user to spread domains through multiple proxies which means

That is fine, but it doesn't meet needs of what I'm describing.

> In fact, remote PACs are not allowed on AMO since there is no control over where the client is being redirected to.

Right, remote PACs have always been user-selectable and not something assigned by an addon (except for the case of WPAD which lives at a well-known address and path).

> In FoxyProxy (or similar) the function to allow users to choose to pass domain A through proxy and not domain B, can be handled
> in PAC and PAC in turn can be handled by the proposal.

You're just missing the point or refusing to see my point. I'm not sure which. And I'm not sure why.

Comment 17

[Eric Jung [:ericjung]]

> You're just missing the point or refusing to see my point

To distill the point even further: if we want the same functionality we had pre-Web Extensions, or we want to support people in school and corporate environments, then addons need to be able to load URLs either by remote PAC *or* through a proxy server selected by the user, depending on URL.

Comment 18

[erosman [:erosman]]

> You're just missing the point or refusing to see my point. 
It is unclear to me.
Do school and corporate environments require users to user their predefined PAC? 
(Personally I have never seen it in universities that I attended or work environments that I worked in so it is new to me)

What is the purpose of that PAC?

What would such a PAC look like?

Are users allowed to use connections other than the aforementioned PAC?

Comment 19

[Eric Jung [:ericjung]]

> Do school and corporate environments require users to user their predefined PAC? 

Yes. HTTP/HTTPS access is blocked if not used through the proxy server. There are usually exceptions (servers hosted inside the intranet can NOT be loaded through the proxy server because they proxy server only talks to the outside world), and so a PAC is used to manage that. Here's a sample with a most of the IP addresses redacted:

function FindProyForURL(url, host) {

  var PROXY = 'PROXY corporate.jak-proxy.xerox8:8080; PROXY corporate.nm-proxy.xerox9:8080';

  if (isPlainHostName(host) && !localHostOrDomainIs(host, "*xerox")) return 'DIRECT';
  else if (shExpMatch(host, "*xerox.") || shExpMatch(url, "*remote.xerox.com*")) return 'DIRECT';
  else if (shExpMatch(host, "*xerox") && !localHostOrDomainIs(host, "proxyid.xerox")) return 'DIRECT';
  else if (shExpMatch(host, "*.foo.com") || shExpMatch(host, "*bar.net")) || shExpMatch(host, "*baz.com"))
		return 'DIRECT';

  var net = dnsResolve(host);

  /* No proxy for internal non-routable DMZ systems */

	if ((isInNet(net, "192.168.0.0", "255.255.0.0") ||
		isInNet(net, "xxx.xxx.0.0", "255.255.0.0") ||
		isInNet(net, "xxx.xxx.0.0", "255.255.0.0") ||
		isInNet(net, "xxx.xxx.xxx.xxx", "255.255.255.240") ||
		isInNet(net, "xxx.xxx.xxx.0", "255.255.255.0") ||
		isInNet(net, "127.0.0.0", "255.255.255.0") ||
		isInNet(net, "0.0.0.0", "255.255.255.255") ||
		isInNet(net, "10.0.0.0", "255.0.0.0") ||
		isInNet(net, "172.xxx.xxx.xxx", "255.240.0.0")) &&
		!localHostOrDomainIs(host, "proxyid.xerox"))
      return 'DIRECT';


	/* No proxy for DMZ systems */

  else if (isInNet(net, "xxx.xxx.xxx.0", "255.255.255.0") ||
		isInNet(net, "xxx.xxx.xxx.0", "255.255.255.0") ||
		isInNet(net, "xxx.xxx.xxx.xxx", "255.255.252.0") ||
		isInNet(net, "xxx.xxx.xxx.xxx", "255.255.255.0") ||
		isInNet(net, "xxx.xxx.xxx.xxx", "255.255.248.0") ||
		isInNet(net, "xxx.xxx.xxx.xxx", "255.255.240.0") ||
		isInNet(net, "xxx.xxx.xxx.xxx", "255.255.254.0") ||
		isInNet(net, "xxx.xxx.xxx.xxx", "255.255.255.0") ||
		isInNet(net, "xxx.xxx.xxx.xxx", "255.255.255.0")  ||
		isInNet(net, "xxx.xxx.xxx.xxx", "255.255.255.0") ||
		isInNet(net, "xxx.xxx.xxx.xxx", "255.255.255.0") ||
		isInNet(net, "xxx.xxx.xxx.xxx", "255.255.255.0")||
		isInNet(net, "xxx.xxx.xxx.xxx", "255.255.255.0") ||
		isInNet(net, "xxx.xxx.xxx.21", "255.255.255.255") ||
		isInNet(net, "xxx.xxx.xxx.xxx", "255.255.255.0"))
      return 'DIRECT';

	else return PROXY;
}

Those proxy servers block sites like facebook, etc. The only way around it is to use a proxy server (typically SOCKS or tor, but sometimes SSL proxy servers can work depending on how the firewall blocks) outside the intranet. Now maybe you can see the situation:

1. facebook and blocked URLs: use my custom proxy server
2. everything else? use the PAC file that is deployed to every machine by the system admins

Comment 20

[erosman [:erosman]]

So far, this is what I have gathered...

Firefox has 2 main methods of proxy implementation:
1- Manual proxy configuration (Proxy:Port)
2- Automatic proxy configuration URL (PAC)

Aforementioned methods are distinct and have separate processes and APIs. 
This proposal only refers to the second method which is PAC Implementation.
It has no bearing on the first method. 

Users can be divided in two main groups:
1- General majority who have access to 'Firefox Connection Settings'
2- Specific group who access using work/education computers and whose access to 'Firefox Connection Settings' has been prevented.

It seems Eric's concern is centred on the second group. It also appears that the group is able to install add-ons despite the restrictions and the intention is to use the add-on to bypass the imposed restrictions by circumventing the 'Firefox Connection Settings' and gaining direct access to the Proxy API.

The proposal in this topic affects majority of the users who are using their own or unrestricted computers. Furthermore, there is no conflict in this proposal with the implementation of a Proxy API (first method). The two methods are not mutually exclusive. Legacy add-ons had a similar setup. Some add-ons based their function on the Proxy API, some on the PAC, and some on a combination of both.

Comment 21

[Eric Jung [:ericjung]]

> 2- Specific group who access using work/education computers and whose access to 'Firefox Connection Settings' has been prevented.

It is not prevented.

> to bypass the imposed restrictions by circumventing the 'Firefox Connection Settings'

I never mentioned that window is restricted.

> there is no conflict in this proposal with the implementation of a Proxy API (first method). The two methods are not mutually exclusive

The problem is that it's extremely unlikely the addons team is not going to implement both because of both human resource limits and philosophy (why should a system like proxying be available through two different methods?). The original proposal covers all use cases while this one does not.

Comment 22

[erosman [:erosman]]

> The original proposal covers all use cases while this one does not.

It does not have PAC functions and a PAC without PAC functions is not really a PAC.
As stated by Chris, there wont ever implement PAC functions and hence the need for an alternative approach ... bug 1353510

Comment 23

[Shane Caraveo (:mixedpuppy)]

proxy config has landed via bug 1425535.  There was some irc conversation between myself and erosman where he feels a pac reload must happen after setting the pref.  I think his analysis is wrong, the code starting here[1] should result in the pac thread being restarted and the new config being loaded.

For any other options (related directly to the prefs) that are not specifically handled we can file follow-up bugs.  It's unlikely we're going to support data uris or blobs for the prefs, that would require low level changes.  I think those issues can be resolved after addressing Bug 1152332, at which point we can support an onProxy event that the background script could handle.

[1]https://searchfox.org/mozilla-central/source/netwerk/base/nsProtocolProxyService.cpp#789