28998 - (wpad) Proxy: Web Proxy Auto-Discovery Protocol (WPAD)

Status: RESOLVED FIXED

(Core :: Networking, enhancement, P3)

Description

[Dax kelson]

Description is available at: http://www.wrec.org/Drafts/draft-ietf-wrec-wpad-
01.txt

Short desc: Allows browsers to automatically locate a proxy-auto-config (PAC) 
file.

Please support at least the "Well Known Alias" discovery mechanism.  IE 5+ 
supports WPAD, and is *very* usefull for sysadmins.  Please have this turned on 
by default.  It should be pretty easy to implement, here is my take.

In order for machine "snoopy.mis.mycompany.com" to find it's PAC file, it tries 
these URLs to find the PAC.

if DNSnameExists(wpad.mis.mycompany.com) 
    try http://wpad.mis.mycompany.com/wpad.dat
elseif DNSnameExists(wpad.mycompany.com)
    try http://wpad.mycompany.com/wpad.dat
else
    no proxy

All a company needs to do is create a DNS A or CNAME record for "wpad" pointed 
at their webserver and create a "wpad.dat" file which is their PAC file 
(setting the MIME type appropriately on the webserver for wpad.dat).

Again, please have this be the *DEFAULT* behavior.

Comment 1

[Sean Richardson]

Not sure: this will be either dependent on, or a DUP of, bug 22404, 
"automatic proxy selection does not work", M15.

Comment 2

[Gagan]

->nobody

Comment 3

[Niko Pavlicek]

i thought it was implemented a few weeks ago, wasn't it?

Comment 4

[Fabian Guisset]

Reporter, is this fixed? Please also see 
http://www.mozilla.org/docs/netlib/pac.html
Can someone get the bug off [e-mail]'s hands?

Comment 5

[Dax kelson]

No it isn't fixed yet.  Although there is no support for PAC files. 
Mozilla/Netscape must be told where to find the PAC via a URL.

WPAD is the mechanism for the browser to automatically get a PAC file (ie, you
don't have to tell the browser the URL of the PAC).

Comment 6

[Dax kelson]

Oops, typo.  I meant to say "Although there is NOW support..."

Comment 7

[benc]

qa to me.

Depends on PAC working (which has moved through three bugs...)

Comment 8

[benc]

After the next big Netscape release, I will approach the various marketing and
product groups that might consider taking this feature, now that PAC is somewhat
working.

If none of these groups feels that Netscape/AOL will add this feature, hopefully
someone in the community will take interest.

Questions:
1- What is the status of this draft? A list of products, and the apparent value
compliance adds is needed.
2- What specific compliance class should we implement?
3- How does this feature interact with other features (such as IPv6?)
4- What are the post-coding requirements? Do we need to configure a DNS domain?
Do we need test servers? How much documentation is needed?
5- Should this be implemented as some kind of module that is not hard-coded in
all builds?

Comment 9

[Peter Murray]

Since no one else has piped up in favor of this, I thought I'd offer a little bit:

> 1- What is the status of this draft? A list of products, and the apparent 
value
> compliance adds is needed.

To the best of my knowledge, the original draft has expired without further 
work.  There is new development in a different working group at:

  <http://www.wrec.org/Drafts/draft-cooper-webi-wpad-00.txt>

There is also some further discussion about the downsides of WPAD and 
how it could be replaced by something called IDD:

  <http://www.wrec.org/webi-archive/200101/0054.html>
  <http://www.wrec.org/webi-archive/>
  
Microsoft continues to include it in MSIE; one presumes they intend to 
continue.  I'm not aware of WPAD's existence in other browsers.

> 2- What specific compliance class should we implement?

Personally, I think class 0 is enough, but I don't know the intent of the 
original authors.

> 3- How does this feature interact with other features (such as IPv6?)

It presumes a proxy autoconfig processor, but other than that...

Comment 10

[Alex Blewitt]

Note that in both the previous and below mentioned drafts of WPAD (note: the 
link http://www.wrec.org/Drafts/draft-cooper-webi-wpad-00.txt has also expired 
as of this comment date), the DNS is not the only way of finding the WPAD file. 
It is also possible to obtain the information via DHCP, which are the two most 
common forms of service available for both corporate and dial-up users.

Those that work in a large organisation will undoubtably have one of DNS or 
DHCP set up, and the appropriate web servers to be able to serve pages. 
Furthermore, those at home (e.g. dial-up users) will almost certainly be using 
DHCP to connect, and WPAD can be made available for them, too.

Given that IE5.x uses it, a large portion of sysadmins are looking forward to 
using WPAD to configure systems browsers. Once this has been implemented in 
Mozilla (and it should be *relatively* trivial to provide the DNS 
implementation, even if not the DHCP implementation immediately) then we will 
be promoting Mozilla as our system-wide browser; however, we're stuck with IE5 
until then.

Comment 11

[bnb]

Dialup acccounts in general get details such as this via PPP options, thus the
DHCP implementation isn't likely to be too much use there, there is however no
reason why the DNS implmentation can't work, once the user has connected.

As someone with a laptop that roams multiple networks I find the lack of support
for this feature particularlly annoying, *** Without it PAC is unusable ***
since you are always having to type in proxy URL's when moving between sites
with different web proxy arrangements, and half the time you don't know what the
URL should be!!

Comment 12

[Patrick McManus [:mcmanus]]

Attachment: lets mozilla use dhcpinform program to use dhcp's wpad option to find pac file

requires dhcpinform program

Comment 13

[Patrick McManus [:mcmanus]]

the patch I uploaded is really just a proof of concept - it lets you put dhcp://
as the value for your proxy pac file and mozilla will use the dhcpinform program
to find the value of dhcp option 252 (wpad) to fill in the real pac location.

dhcpinform (located at http://www.ducksong.com:81/dhcpinform/) is implemented
separately due to permission problems (it needs to bind to a low number port to
implement dhcp), but its GPL'd and can certainly be integrated if someone cares to.

And like I said, this is really just proof of concept.
 

Comment 14

[benc]

Can you make it "X-dhcp:" as the scheme name?

I've declared a blockade against non-RFC complaint scheme names in bug 130089.

Comment 15

[Michael T. Babcock]

Since this is an issue I care about, I'd like to pipe in alongside the original
poster that at the very least, the current IE-implemented protocol be coded in.
 All of my clients use WPAD with Internet Explorer and I have to set up the
proxy settings manually for their Mozilla browsers.

Where host.domain.tld is the machine name, wpad.domain.tld should be checked for
a web server serving the file wpad.dat.  Where domain is made up of multiple
parts (not including .tld), each level should be checked.  Where .tld is made up
of multiple parts (true in some parts of the world), we simply have a bug.

Comment 16

[Michael T. Babcock]

Is it at the very least possible to edit one's own preferences JavaScript file
to specify "http://wpad.{currentdomain}/wpad.dat" as the auto-proxy URL?

Comment 17

[benc]

Preferences | Advanced | Proxies

Comment 18

[Michael T. Babcock]

My reference to {currentdomain} meant that I want it to _automatically_ use the
current domain value from JavaScript; not type it in every time I change domains
(which I do often with my computers).

Comment 19

[benc]

I think that is the reason we have this bug :)

Comment 20

[Juan Lang]

Attachment: allows mozilla to use wpaddetekt program to find pac URL

Supercedes previous patch--changes scheme name from DHCP to X-wpad, and uses
detektor package to get cross-platform support.

Comment 21

[Juan Lang]

Attachment: Package required for mozwpad.patch

Required scripts and utilities to support mozwpad.patch

Comment 22

[Juan Lang]

The patches I submitted work on Linux, FreeBSD, Solaris, MacOSX, and Windows. 
The detektor package has a sourceforge project,
http://sourceforge.net/projects/detektor/

This implements both dhcp-based discovery and DNS lookups for wpad.yourdomain. 
It also tries SRV records, although this is pretty experimental.

Add'l improvements include trying to figure out if you're using DHCP (so far on
Linux, FreeBSD, and Windows), to avoid unnecessary delay if you're not using it,
and removal of buffer overruns.

Comment 23

[Juan Lang]

Attachment: Package required for mozwpad.patch

Fixed SRV lookups in UNIX, and use them in Windows (2000/XP only), now that
I've had a chance to test them.

Comment 24

[Darin Fisher]

-> me

Comment 25

[Darin Fisher]

Comment on attachment 115185 [details] [diff] [review]
allows mozilla to use wpaddetekt program to find pac URL

juan: thanks for the patch.  i'll try to take a look shortly.  one thing that
is definitely bad with this patch is your use of nsIFileSpec.  that interface
is deprecated.	you should use nsIFile instead :)

otherwise, i'll need to spend some time reading up on WPAD and the project on
SF.

thanks for the patch!

Comment 26

[Juan Lang]

-> me.  Thanks Darin.  I'm working on a new patch that does DNS lookups within
Mozilla, and adds a new checkbox to the proxy preferences UI to specify WPAD. 
I'll submit it with nsIFile.

Comment 27

[Juan Lang]

Attachment: New and Improved

Uses nsIFile rather than nsIFileSpec.  DNS lookups handled within Mozilla. 
Always looks for support programs in Moz bin dir.  New radio button in proxy
prefs dialog.

Comment 28

[Michael T. Babcock]

I don't see the wpaddhcp.bat or wpaddhcp programs in this patch ("New and
Improved" from 2003-03-07 00:46:33), is it in a previous one?  Looking at the
patch, I'm curious as to why there's still an #ifdef'd (Windows|Unix) section at
all, but I haven't been in this piece of code before, so it may seem reasonable
to the others involved.

Comment 29

[Juan Lang]

Attachment: Package required for mozwpad.patch

All the required utils are here.  (Bugzilla **** out mid-upload last night,
apologies.)  You'll need the following in your mozilla bin directory:
UNIX:  scripts/detektdhcp scripts/wpaddhcp src/dhcpinform
Windows:  scripts/getdhcp.bat scripts/wpaddhcp.bat src/dhcpinform.exe

The ifdef'd sections are because dhcpinform only works on UNIX and Windows. 
Also, I open a pipe to it, and the pipe facility is platform-specific.

Comment 30

[Juan Lang]

Folks, I've got a dilemma on this bug.  I used proxy type "4" for my new WPAD
proxy type in order not to tromp on bug 115720.  However, the radio group
display code doesn't handle skipping radio button values right, and the pref
doesn't show up as selected.

I had thought of coopting pref 2 as well, thinking this would be a cleaner UI: 
specify "autoconfig" as your radio button pref (type = 2), and have a separate
checkbox if you want to specify your PAC URL rather than use WPAD.  This isn't
backward-compatible with existing prefs files, however.

Suggestions?  Sorry for the spam.

Comment 31

[benc]

(being the person who wrote bug 115720 comment 18...)

I think it is safe to say, as far back as I can think, that you could use three.

Although the default for Communicator was ==3, that value was almost never
written to disk because prefs.js only contained prefs that were not set to the
default. Bugzilla contains 4 reports of this line coming over, but most people
did not have this line in their prefs.js, even if they were running Communicator
w/o proxy (type==3). This is evident for two reasons:

1- We only got 4 reports. If this were a common problem, we would have broken
every migration that was not using manual. (We broke every migration that was
using PAC for a long time (because PAC was broken so we know that there would
have been hundreds of complains if we broke direct (proxy off) user migrations).

2- I checked about a dozen profiles on my old Win98 laptop, and none of them
have type==3. I have one each of type==1 and type==2. This laptop was used to
test proxy configurations, so it is probably more representative of proxy usage
than the average laptop, where all the profiles would be the same setting.

How did those unlucky 4 people get stuck?

What probably happened, and nobody bothered to check, was some people using
Mission Control or other prefs-management schemes did this. There may also be
situations where prefs.js was locked, so the value could not be purged.

Can you put a ASCII-art mockup of how you want to make prefs work? Is WPAD just
a mode selection, or is there additional parameters?

Right now, if we see "3", we set it to "0", which clears the bogus value when
prefs.js is saved. You might be able to use 3, if you have WPAD default behavior
that won't make the browser go deaf if a user accidentally goes into that mode.

You should get comments from Darin, b/c he is module owner.

Comment 32

[Darin Fisher]

i don't think you should muck with the proxytype.  breaking backwards
compatibility with older versions of mozilla is not a good idea if it can be
avoided, and in this case you can avoid the problem by introducing new pref
values.  perhaps WPAD should just be a special case of regular PAC.  then you
can keep the proxy type as is, but add an extra pref to control how PAC is done.

Comment 33

[Juan Lang]

Attachment: Uses proxy type 3

This tromps 115720.  Hoping this isn't an issue, as benc points out that it
isn't common, and it doesn't break compatibility with mozilla.org mozilla, only
with Netscape 4.x, and it requires using the Netscape 4.x prefs file with
mozilla which isn't recommended anyway.  The UI adds the following pref value
to the proxy dialog:

o Automatic proxy configuration
o Specify proxy configuration URL:
  _______________________________ (Reload)

The first radio button shown specifies type 3, aka WPAD.  The second radio
button shown specifies type 2, the existing PAC URL.  Same as existing PAC
radio button, with text changed.

An alternate UI could be:
o Automatic proxy configuration:
  x Automatically detect configuration URL
  Specify configuration URL:
  _________________________ (Reload)
The radio button would specify WPAD or PAC, and the check box would specify
WPAD when enabled.  The default would be disabled, since existing profiles with
type=2 wouldn't have the WPAD option.
I think it's kinda goofy, since the least-user-interaction mode (WPAD) requires
two clicks to enable.  But it's the least risky.

Please advise the preferred approach, and I'll resubmit if necessary.

I also fixed a problem loading WPAD at startup with this patch, and fixed a
couple style problems.

Comment 34

[Juan Lang]

Attachment: More style fixes

Makes a little less noise when WPAD doesn't find a PAC URL.  Replaced NULL with
nsnull.

Comment 35

[Juan Lang]

Attachment: New unified patch

Reimplemented DHCP helper w/necko

Comment 36

[Darin Fisher]

thanks for the updated patch Juan.  since this is not a feature requirement for
mozilla 1.4, it is unlikely that i will have the time to really focus on this
much during the 1.4 cycle.  let's target 1.5 for this.

Comment 37

[Christian :Biesinger (don't email me, ping me on IRC)]

Comment on attachment 115185 [details] [diff] [review]
allows mozilla to use wpaddetekt program to find pac URL

(clearing review request on obsolete patch)

Comment 38

[Christian :Biesinger (don't email me, ping me on IRC)]

Comment on attachment 117812 [details] [diff] [review]
New unified patch

+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License Version 2 or later
+ * (the "GPL").

such code can not be part of mozilla. see
http://www.mozilla.org/MPL/license-policy.html

Comment 39

[Juan Lang]

Oops.  Thanks Christian.  Consider it relicensed (it's all written by me).  I'll
resubmit with appropriate headers one of these days.

Comment 40

[Christian :Biesinger (don't email me, ping me on IRC)]

Comment on attachment 117812 [details] [diff] [review]
New unified patch

+    nsIFile *file = nsnull;

use a COMPtr, it looks like you leak this object.

+    if (!NS_FAILED(rv) && file != nsnull)

if (NS_SUCCEEDED(rv) ...
several times

+	     nsSharableCString wpn;

I am not familiar with this string class, but to me, it looks like you should
rather use nsCAutoString

+    if (wpadURL.Length() == 0) {

if (wpadURL.IsEmpty())

+	     char *resolvedWpadHost = nsnull;

use an nsXPIDLCString, you leak this string.

+    if (wpadURL.Length() > 0)

if (!wpadURL.IsEmpty())

in pref-proxies.js, the code in the two case labels is almost the same, can't
you share it somehow? e.g. set the disabled attribute on auto outside the
switch and then use:
    case "2":
   auto[i].removeAttribute( "disabled" );

    case "3":
     default:
       for (i = 0; i < manual.length; i++)
	 manual[i].setAttribute( "disabled", "true" );
       if (!radiogroup.disabled)
	 for (i = 0; i < auto.length; i++)

       break;

(untested)

Comment 41

[Juan Lang]

Attachment: Updated with Christian's comments

Fixed license headers, and fixed leaks (thanks Christian!)

Christian, on pref-proxies.js:	what you suggest doesn't actually reduce lines
of code.  I'm leaving this patch alone to lessen the size of the patch.  I have
another version I'll post shortly which may be more readable overall, please
give your feedback on that one.

Comment 42

[Juan Lang]

Attachment: Readability enhancement for pref-proxies.js

Christian, try this one on for size.  Makes DoEnabling() more readable, I
think.	 Not functionally any different than the one in the patch.

Comment 43

[Juan Lang]

Attachment: Updated to 1.4

Keeping up-to-date with 1.4.  Also separated new files into a tarball I'll post
separately.  I read somewhere diffing against /dev/null was the preferred way,
but actually applying this sort of patch was too ugly for me.

Comment 44

[Juan Lang]

Attachment: New files

Reimplemented DHCP stuff as an XPCOM component, and do it in-process under
Windows.

Comment 45

[Juan Lang]

Attachment: Small cleanups

Some style nits..

Comment 46

[Juan Lang]

Attachment: Updated files

and include cleanups, to save the reviewers a little work.

Comment 47

[Juan Lang]

Comment on attachment 119388 [details] [diff] [review]
Updated with Christian's comments

Clearing review request on obsoleted patch

Comment 48

[Juan Lang]

Attachment: Remerged w/trunk

Comment 49

[Juan Lang]

Attachment: Minor comment improvements

Comment 50

[Juan Lang]

Attachment: NT4 support

and untested support for non-Ethernet hardware.

Comment 51

[Darin Fisher]

ok, here's what needs to be done with this patch:

1) the UI changes need to be broken out into a separate patch.  the backend
changes can be addressed first with prefs to control what is done.

2) let's start with just getting the necessary hooks in place inside necko so
that a WPAD extension can be possible.

then, create a WPAD extension... maybe start by hosting it on mozdev.org.  if it
proves successful, and enough users ask for this to be incorporated into mozilla
proper, then we can look into doing that.

otherwise, i think this patch is going to continue to stall.  i'm sorry i've
been so slow in reviewing this patch.

Comment 52

[Robert Accettura [:raccettura]]

1.  Is this platform independant or W32 only?

2.  Any chance on 1.7a?  This would be a major stride in helping the marketing
effort believe Mozilla/Firebird is worthy of office use.  Many offices require
such capabilities.  Bringing this to the table means a smoother transition in
the eyes of IT departments considering the switch.

Comment 53

[Jo Hermans]

Can we at least implement the DNS-part of the implementation ? That should be
pretty easy to implement (even in-Mozilla, it doesn't need an extension), and XP
too. Most 

SRV-lookups are much more complicated, and we don't support it anywhere else
(see bug 14328). DHCP-lookups are the most complicated of all, and it requires
the external dhcpinform.exe program (which will be suid-root !). I doubt that it
will be XP, and I don't think it has much chance to be included with the trunk,
but an extension will be possible.

In my company, I'm involved with quite a few very large dialup and broadband
access networks in the world. I have never seen SRV- and DHCP-based PAC-lookups
in those networks, which are a mix of public access and vlan's, but I have seen
a few DNS-based lookups. They're mostly used in in-house networks, like for
office intranets. It seems to be a feature that nobody knows about, because
Internet Exploiter does all this magic behind the scenes (it's a sign that the
sysadmin knows his job).

Comment 54

[Juan Lang]

Folks, pls read the comments before posting.  Yes, it's x-plat, and yes, it
supports dhcp and dns well-known-names.  (I removed SRV lookup support to stick
with what necko supports.)  If you have time to rework it into an extension,
have a go, the code is all here.

Comment 55

[Darin Fisher]

*** Bug 229767 has been marked as a duplicate of this bug. ***

Comment 56

[David Hallowell]

Nominating to block aviary 1.0 as this feature is almost essential for anyone
downloading firefox in a corporate environment that uses wpad and has either a
clueless or unhelpful helpdesk.

e.g. where I currently work they use proxy auto config, back when I started I
downloaded Firebird 0.6 and it didn't work - I noticed IE had auto proxy config
checked. I tried contacting the helpdesk for the proxy info and their response
was "we stopped supporting Netscape in 2000 and only support IE now" (note: I
never said I was using Netscape). Anyway I worked it out myself looking at this
bug (the auto config url was http://wpad.xxxx.co.uk/wpad.dat

When Firefox 0.9 came out the press coverage made a lot of people in the office
download Firefox, it imported the settings from IE as expected but becaue WPAD
was used they couldn't connect to the network. Therefore a lot of people were
put off because it appeared not to work. I could help the people in my office
that were vocal enough, but we have a lot more offices and I can't do anything
about them as the helpdesk don't want to be educated.

Comment 57

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

I agree that we should try to get it in Fx 1.0, and I'll test the patch against
aviary shortly.  (Note that I'm _not_ marking it blocking1.0+ at this time.)

Comment 58

[Brendan Eich [:brendan]]

Marking blocking.  Importing IE settings implies supporting WPAD, and we don't
want to lose the easy migration for IE users.  We just need this patch.  Cc'ing
darin, who really should have been cc'd already.

/be

Comment 59

[Brendan Eich [:brendan]]

I'm blind in the morning without caffeine, never mind me (but feel free to drop
a quarter in my empty coffee cup :-).

/be

Comment 60

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

I don't think the call to dnsService to look up "wpad" will do what we want
here.  It will check the search path, which is not necessarily the same as the
suffix of the machine's FQDN.  Can anyone say, preferably with an
authoritative-ish doc link or results of testing, what IE does when the DNS
search path and FQDN-suffix disagree? 

Comment 61

[Nick Hatch]

IE uses the FQDN for the proxy auto-discovery. I don't believe that it uses the
search path at all. (MS had a little oopsie in regards to their first wpad
implementation, this page has a slightly more technical description than their
other pages)
http://www.microsoft.com/technet/security/bulletin/ms99-054.mspx

The IETF draft on Web Proxy Auto-Discovery also specifies that the FQDN should
be used, with no mention of the search path.

Comment 62

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

Thanks, Nick, that's exactly the information I was after.

I think we want to only use the single stage of FQDN substitution, rather than
risk that bug again.  Patch coming up this weekend with that change effected.

Comment 63

[Roy Badami]

If you don't walk up the tree, that's a departure from the WPAD spec, and that
means it won't work out-of-the-box for all users.

Unfortunately implementing the spec safely requires knowledge of the structure
of all TLDs, so as to know when wpad at second level is within the same
administrative domain, and when it's not.  And even then, there are problems
with private registries (what happens if someone registers wpad.uk.com?)

So it's probably quite reasonable to depart from the spec.  In which case using
the search path and just resolving the name 'wpad' seems quite a reasonable
compromise...

Whatever happens, if Mozilla departs from the WPAD spec, then its behaviour
needs to be prominently documented in the release notes...

-roy

Comment 64

[chris hofmann]

shaver, still thinking this will make RC2 coming up soon?

Comment 65

[chris hofmann]

shaver, still thinking this will make RC1, which is coming up soon?

Comment 66

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

Attachment: aviary: back-end, WKS only

This is _just_ the WKS version of WPAD (http://wpad/wpad.dat).	I spoke to some
of  the net-config folks at OLS this week, and I think we can get the dhcp
value without an external program.  I'll work on that separately, and look at
the NT4 dhcp support as well, but I think this covers the vast majority of
autoconfig cases.

Need a UI patch for firefox as well, coming up separately.

Comment 67

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

Comment on attachment 154294 [details] [diff] [review]
aviary: back-end, WKS only

Darin, baby, gimme some love on this minimal little beauty.

Please ignore the tab-damage in the nsProxyAutoConfig.js changes; I'm breaking
in a new xemacs on this laptop.

Comment 68

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

Attachment: aviary: pref-connection.{xul,js,dtd} fixes

Just a nice, simple patch, since I suppressed the urge to rewrite
pref-connection.js for reuse and codesize.

Comment 69

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

Still to do:

 - update (Win) IE migration code to set up prefs correctly on first migration.
 - figure out helper-less DHCP situation for Unix and Windows

Comment 70

[Christian :Biesinger (don't email me, ping me on IRC)]

+    nsCAutoString wpadURL("http://wpad/wpad.dat");

NS_NAMED_LITERAL_CSTRING?

Comment 71

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

Aye, duh.  Churned that code too many times, thanks.

Comment 72

[Darin Fisher]

Comment on attachment 154294 [details] [diff] [review]
aviary: back-end, WKS only

>Index: netwerk/base/src/nsProtocolProxyService.cpp

>+static NS_DEFINE_CID(kDNSServiceCID, NS_DNSSERVICE_CID);
>+
>+void
>+nsProtocolProxyService::ConfigureFromWPAD(void)
>+{
>+    nsCOMPtr<nsIDNSService> dnsService = do_GetService(kDNSServiceCID);

nit: use ContractID instead of ClassID


>+    if (NS_FAILED(dnsService->Resolve(NS_LITERAL_CSTRING("wpad"),
...
>+        ConfigureFromPAC(wpadURL.get());

hmm... ConfigureFromPAC dispatches a PLEvent to do its work asynchronously.
couldn't we use the Async host resolver then?  I think that would be better
than blocking the UI thread to resolve "wpad"

use AsyncResolve and then call ConfigureFromPAC from your nsIDNSListener's
onLookupComplete method.


>Index: netwerk/base/src/nsProxyAutoConfig.js

>         if (!this.done)
>             return null;
>+		
>+		if (!LocalFindProxyForURL)
>+			return null;

nit: convert tabs to spaces.


>-        return LocalFindProxyForURL(uri.spec, uri.host);
>+		return LocalFindProxyForURL(uri.spec, uri.host);

nit: convert tabs to spaces.

Comment 73

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

Attachment: aviary: back-end, WKS only, AsyncResolve

Now with AsyncResolve _and_ actual working WPAD use.  WFM!

Should have fixed all the tab-damage, too.

Comment 74

[Darin Fisher]

Comment on attachment 154587 [details] [diff] [review]
aviary: back-end, WKS only, AsyncResolve

>Index: netwerk/base/src/nsProtocolProxyService.cpp

>+    nsCOMPtr<nsIDNSRequest> request;
>+    (void)dnsService->AsyncResolve(NS_LITERAL_CSTRING("wpad"), PR_TRUE, this, curQ,
>+                                   getter_AddRefs(request));
>+}

nit: spit out an error of some sort if AsyncResolve should fail?


>+
>+nsresult
>+nsProtocolProxyService::OnLookupComplete(nsIDNSRequest *aRequest,
>+                                         nsIDNSRecord *aRecord,
>+                                         nsresult aStatus)
>+{
>+    nsCAutoString wpadURL("http://wpad/wpad.dat");
>+    if (aRecord != nsnull)
>+        ConfigureFromPAC(wpadURL.get());
>+}

this won't compile without |return NS_OK;| on some compilers.

also, I'd use

  const char wpadURL[] = "http://wpad/wpad.dat";

instead of the nsCAutoString, or I might even just hard-code the
string literal in the call to ConfigureFromPAC ;-)


r=darin with those nits picked.

Comment 75

[Roy Badami]

What was the decision on removing domain components as per the WPAD spec?

Does it just prepend wpad to the domain, and give up if that doesn't work?  Does
it try to resolve wpad using the resolver search path?  Or does it implement the
spec (with suitable mitigation for the security risks)?

If the Mozilla implementation differs from the WPAD spec then this needs to be
explained in the release notes, so that we sysadmins understand the implications...

-roy

Comment 76

[Christian :Biesinger (don't email me, ping me on IRC)]

Comment on attachment 154587 [details] [diff] [review]
aviary: back-end, WKS only, AsyncResolve

some style comments :)

+nsProtocolProxyService::ConfigureFromWPAD(void)

style of this file is () instead of (void)

+    /* Asynchronously resolve "wpad", configuring from PAC if we find one (in
our

style of this file is also // even for multiline comments

+    if (aRecord != nsnull)

style of this file is if (aRecord)

Comment 77

[Christian :Biesinger (don't email me, ping me on IRC)]

>+nsresult
>+nsProtocolProxyService::OnLookupComplete(nsIDNSRequest *aRequest,

doesn't this need to be NS_IMETHODIMP, for windows?

Comment 78

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

Attachment: aviary: back-end, simple WKS only, nits picked

This is the patch I'm committing shortly to aviary; thanks for the style and
nit catches, all.

Roy: this comment from the patch describes how we diverge from the WPAD spec. 
Would it be suitable as release-note fodder?

    // We diverge from the WPAD spec here in that we don't walk the hosts's
    // FQDN, stripping components until we hit a TLD.  Doing so is dangerous in

    // the face of an incomplete list of TLDs, and TLDs get added over time. 
We
    // could consider doing only a single substitution of the first component,
    // if that proves to help compatibility.

Comment 79

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

I still need (someone) to port the backend patch to trunk, and probably the
pref-UI one as well.  I'm not sure if I'll get to that before devday; if I have
more time for WPAD hacking it'll likely be spent in this order:

 - IE migration support
 - simple first-component substitution of FQDN
 - investigating DHCP for Windows and modern-Linux, sans helper app

Comment 80

[Christian :Biesinger (don't email me, ping me on IRC)]

Attachment: trunk: back-end, simple wks only (checked in)

trunk version of attachment 154643 [details] [diff] [review]. I also moved an nsresult rv; to where it is
first used.

Comment 81

[Darin Fisher]

Comment on attachment 154657 [details] [diff] [review]
trunk: back-end, simple wks only (checked in)

Thanks biesi!

Comment 82

[Christian :Biesinger (don't email me, ping me on IRC)]

Comment on attachment 154657 [details] [diff] [review]
trunk: back-end, simple wks only (checked in)

trunk backend checked in

Comment 83

[benc]

Worked my way through the IETF draft and bug comments today...

Should we get a separate bug for the pref UI? I have comments on
network.proxy.type==4, but I'll wait for a decision...

The IE logic for domains is not clearly documented by Microsoft:

http://www.microsoft.com/technet/security/bulletin/fq99-054.mspx
"
How does IE 5.01 eliminate the vulnerability?
IE 5.01 has a more robust search algorithm that does not blindly search
third-level domains. Instead, it searches third-level domains only when appropriate.
"

I'm not sure why we add all the DNS stuff like:

+    rv = dnsService->AsyncResolve(NS_LITERAL_CSTRING("wpad"), PR_TRUE, this, curQ,
+                                  getter_AddRefs(request));

We only support doing A rec lookups right? So why don't we just set the PAC URL
and load it, it is going to lookup "wpad" when the URL tries to connect. If a
PAC load fails, then we use "DIRECT" always anyhow.

Finally, I'm curious if using the hostname-only is really going to get us the
DNS search order we want. The draft is pretty clear about using the "wpad" + the
 default domain of the system, and then moving up the DNS tree. That may not be
the same as blindly calling for "wpad".

Comment 84

[Stefan Borggraefe]

Is someone already working on a frontend-patch for Seamonkey?

If not, I could contribute this part.

Comment 85

[Dax kelson]

(In reply to comment #83)
> Worked my way through the IETF draft and bug comments today...
> 
> Should we get a separate bug for the pref UI? I have comments on
> network.proxy.type==4, but I'll wait for a decision...
> 
> The IE logic for domains is not clearly documented by Microsoft:
> 
> http://www.microsoft.com/technet/security/bulletin/fq99-054.mspx
> "
> How does IE 5.01 eliminate the vulnerability?
> IE 5.01 has a more robust search algorithm that does not blindly search
> third-level domains. Instead, it searches third-level domains only when
appropriate.
> "

How about if mozilla had a white list?

For example, if the top level domian portion of the host's FQDN is: com, net,
org, edu, mil THEN go ahead and search down to the third-level domain level,
otherwise stop at the forth level.

This would be a conservative safe choice, while still following the spec, and
the white list could be expanded later.

Comment 86

[Jo Hermans]

(In reply to comment #85)
> How about if mozilla had a white list?
> 
> For example, if the top level domian portion of the host's FQDN is: com, net,
> org, edu, mil THEN go ahead and search down to the third-level domain level,
> otherwise stop at the forth level.
> 
> This would be a conservative safe choice, while still following the spec, and
> the white list could be expanded later.

So, (following the same examples at the MS-document) if you had a client at
a.b.microsoft.com, you would search wpad.b.microsoft.com and wpad.microsoft.com.
But if you had a client at a.xyz.com.au, you would only search wpad.xyz.com.au ?
That might work fine for TLD's like Australia and the UK, but not for countries
that don't insert second-level doamins like *.com.* or *.co.*, like Belgium. For
example, a Belgian client like a.b.xyz.be would only look for wpad.b.xyz.be, not
wpad.xyz.be ! But for academic institutions, it would be ok, because *.ac.be is
really used (and is probably the only second level domain used here).

It's not just com, org, net or gov ! The UK uses for example ac, co, ltd, me,
plc, sch, nic, nhs, police and mod. Isarel also has k12, China has a whole
bunch, etc ... This might become a very large whitelist. See the list at
<http://www.neuhaus.com/domaincheck/domain_list.htm>, although it's not complete
(ac.be and nic.uk are missing f.i.).

Is there no better solution ?

Comment 87

[Dax kelson]

(In reply to comment #86)
> This might become a very large whitelist. See the list at
> <http://www.neuhaus.com/domaincheck/domain_list.htm>, although it's not complete
> (ac.be and nic.uk are missing f.i.).
> 
> Is there no better solution ?

Is it that bad really? The white list only has the lists the ones that should be
searched to the 3rd level. Everything else stops searching at the forth level.

According to that link, there are 252 domains that are OK to search to the 3rd
level. Take a gradual approach to expanding the white list.

If something doesn't make the whitelist, it will still be searched to the 4th
level which is better than the current solution of just looking for "wpad".

Later the white list can easily be expanded.

Comment 88

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

I'd happily review a patch to use a whitelist (preferably one that can be stored
in a resource file outside compiled code, for easier updating in the field) for
additional checks, but I'm not going to get to it myself any time soon.  The
current code works for most (virtually all?) installations, and I'm strapped for
time in the 1.0 schedule anyway.

Note that it must be the case that failing to update the WPAD domain whitelist
causes WPAD to fail in a case that it might otherwise succeed, and not open us
up to wpad.com-style attacks when a new TLD or sub-TLD is brought online.

Comment 89

[benc]

Whitelists create their own problems, but I do like shaver's proposed update system.

I'm still wondering how much microsoft compatability we can maintain if they
don't tell us exactly what they do.

I also would like to get a technical response to my comment about using a
straight PAC load vs. DNSlookup.

Comment 90

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

I'm going to mark this fixed, based on the trunk commit and lack of subsequent
deployment-woe reports.

Please file additional bugs on specific enhancements to the lookup mechanism
that you'd like to provide.

Comment 91

[benc]

Filed bug 264002 and bug 264632.

Comment 92

[Dan Nelson]

Shouldn't network.proxy.type default to 4?  If the user has to manually set it
to 4, they might as well just set the autoconfig URL themseves :)  The idea of
WPAD is to make a fresh mozilla install use a local proxy without the user
having to do anything.

Comment 93

[Dax kelson]

(In reply to comment #92)
> Shouldn't network.proxy.type default to 4?  If the user has to manually set it
> to 4, they might as well just set the autoconfig URL themseves :)  The idea of
> WPAD is to make a fresh mozilla install use a local proxy without the user
> having to do anything.
>

I think it should default to 4.

Here is what IE does...

If the computer IP address is obtained via DHCP, or PPP then IE does WPAD by
default. If the computer has a statically assigned IP address, then IE doesn't
do WPAD by default.

Comment 94

[Darin Fisher]

> Shouldn't network.proxy.type default to 4?  If the user has to manually set it
> to 4, they might as well just set the autoconfig URL themseves :)  The idea of
> WPAD is to make a fresh mozilla install use a local proxy without the user
> having to do anything.

Yes, I agree.  Our WPAD implementation is pretty silly without this pref change.

Comment 95

[Joseph William Baker]

There is a site wpad.com  which gets a huge amount of traffic because some client machines apparently don't have a localdomin configured.
I would like to see the auto-detect configuration enabled by default but there should be a check to insure that there is a local dns domain configured first.  If not the browser should fall back to "direct" connect mode for the current session.  I'm hopeful that such a quick dns check wouldn't break anything.

Comment 96

[Lee Carré]

I'm a sys-admin, does anyone know which version of firefox will use DHCP for WPAD? if there is existing information available, please point me to it

Comment 97

[Jo Hermans]

(In reply to comment #96)
> I'm a sys-admin, does anyone know which version of firefox will use DHCP for
> WPAD? if there is existing information available, please point me to it
> 

No official version of Firefox uses DHCP for WPAD (attachment 121707 [details] [diff] [review] was never checked in). The only thing what currently happens, is that FF will look for <http://wpad/wpad.dat> if "auto-detect proxy-settings for this network" is selected.