44980 - Proxy: fix no proxy so host aliases can match the full domain

Status: RESOLVED WONTFIX

(Core :: Networking, defect, P3)

Description

[justin]

For many internal intranet sites, it's common to use the host alias when
entering URLs. For example, if I was in the bar.com domain, and I wanted
to access the host foo.bar.com, I would usually just type http://foo/ to get
there.

However, when sitting behind a proxy, there are problems doing this, since the
no_proxies_on list is for the domain (ie. we don't proxy on bar.com). That makes 
mozilla think http://foo/ is on the other side of the firewall.

Also, Netscape 4.x didn't mind if the no_proxies_on domain included a star in 
front (ie. *.bar.com). mozilla doesn't get rid of that, and then nothing 
matches.

Anyway, I've fixed both of these with some changes to nsProtocolProxyService.cpp
(btw, the following patch was made in mozilla/netwerk/base/src).

For the second part, I just strip the star off the front of any text added to 
the no proxy list.

For the first part, if a host doesn't hit anything in the list unchanged, and 
that host does not contain a dot (ie. foo but not foo.com), then I resolve the 
name and grab the canonical host name. I go through the no proxy list again, and 
see if anything hits it.

Also, to avoid having to resolve these aliases every time, if a FQDN version of 
an alias matches something in the no proxy list, then I add the alias to the no 
proxy list. That way, future proxy checks will hit the alias directly, avoiding 
the DNS lookup for all future accesses.

Comment 1

[justin]

Attachment: Patch for this bug

Comment 2

[Andreas Franke (gone)]

Seems reasonable to me. Setting to NEW, adding patch keyword.

Comment 3

[Gagan]

two things--

for a * processing you need more than just strip it off since I could want to 
make things like ad.*.com or even better ad*.*.com So there is more stuff 
involved here than probably possible for this release. 

the remaining fix seems reasonable though I'd prefer to look over it again 
before calling it reviewed. thanks!

Comment 4

[justin]

I agree that the wildcard processing could be better, but the 4.x browsers did 
support (at least) "*.bar.com". It's been my experience that most people try 
typing in the local domain with the star prefix and expect that to work. Since 
it's trivial to strip it off, I think that bit should go in for the sake of 
compatibility.

Comment 5

[Gagan]

I would prefer to build the list of no_proxies_for as a FQDNs that way we don't 
run into doing PR_GetHostByName again and again. Anyway I think we can relnote 
this for beta3 and so giving it a nsbeta3- (as much as I would like this to be a 
plus, we have to leave this out for other beta3 nominations)

Comment 6

[benc]

mass move, v2.
qa to me.

Comment 7

[benc]

I think the underlying problem is that we should take on hostname -> fqdn
conversion ourselves. I think we also need to support display of the systems
default domain and reconfiguration of it to a profile or selectable value (bug
88217). 

The main reason why is this: a user with a hostname is generally on an intranet,
and they generally know what that intranet is. By implication, that means that,
although the URL is deterministically mean to refer to a URI on a certain
service socket. The problem is that when you hand of a hostname to name
resolution, you can't predict where the resolution process will go. This varies
by OS, resolution methods involved (files, NIS, DNS), and system configuration. 

Mozilla is built across numerous operating systems, which potentially have
different ways of handling search domains, hostname or non-fqdn completion.
Vendors have changed search algorithms over time, and it is likely that they
will continue to do so. The hostname completion and search features were
originally designed to allow users to be lazy in typing, while searching their
subdomains. The vast majority of domains are now flat or delegate one level. 

Nobody should be so lazy as to create URL's that have just a hostname (but we
all are...), but what we need to do is make a browser that allows the user to
control this, because increasingly, they will not be able to reconfigure their
OS resolver (because of access control or user sophistication limitations)...

If we did this completion BEFORE we did the no-proxy check, then this problem
would be solved.

Comment 8

[Keith Briscoe]

I'd increase the priority on this bug, as it has bad interactions with other bugs.  For example, if I want to avoid proxying http://foo/, I can try to exclude ".bar.com" (which doesn't work due to this bug), but I also could try to exclude "123.45.", which also does not seem to work.  I would be quite happy being able to wildcard an entire class B (or C) subnet, and a lot of corporate customers would too.  I appreciate that all good web apps should use FQDNs, but they really do not.  In fact, many networks rely on host aliases for easily-guessable web app names (e.g. "timecard", etc).  If we can't figure out the FQDN from the host alias prior to the proxy check, could we at least have the ability to do it with numerical addresses?

Comment 9

[benc]

some cleanup -> sending to eng triage

Comment 10

[Doug Turner (:dougt)]

-> gordon

Comment 11

[rkc]

This bug, more than any other bug, is why I cannot use Mozilla as my default
browser.  Our IT department assumes we are on the intranet, and has many
IT-provided pages that are referenced without full names.  Other browsers that
are more commonly used (IE) perform the auto-FQDN and direct requests within the
intranet or to the proxy server respectively.

The priority and severity of this bug should be increased significantly.

Comment 12

[benc]

+clean report.

I was working on updating bug 72444, and I was searching for the bug that says
that hostname needs to be mapped to an FQDN w/ the default domain name so that
users can block unqualified requests by putting the default domain into "no
proxy for". In the past, I have had trouble finding that bug, because it seems
like there were several bugs that described the same problem and had their own
dupe lists. I think that bug 91587 is the right bug for this, so I'm morphing
this bug to the match the patch justin put in here.

Having 4xp compatability is very important, because people do migrate their
profiles, and as #4 states, they will have "*"s in their profile. I just tested
this, and found that "*" no proxies do not work, which would break connectivity
in an environment where proxies refuse to forward to systems in the no-proxy list.

Gagan's concerns in #3 are useful on a theoretical basis, but I have never heard
of anyone doing that with Communicator, and my personal contacts in Netscape and
iPlanet support suggest this never worked, and/or they would not support it anyhow.

That being said, this patch, absent bit-rot concerns, should probably be checked in.

Comment 13

[benc]

*** Bug 31284 has been marked as a duplicate of this bug. ***

Comment 14

[R.K.Aa.]

how does this relate to bug 48718?

Comment 15

[benc]

These two could be duplicates, depending on how you frame the details.

Comment 16

[Alexander Bashkirov]

Also it would be good if in "no proxy for" field mozilla supported maskin like
192.168.0.*

Comment 17

[benc]

we have bug 80918 for that. please search for a specific bug rather than put a
comment into this bug that causes drift.

Comment 18

[Samir Gehani]

adt: nsbeta1-

Comment 19

[nick]

is this bug still being looked at?

its still present in Firefox 0.9 and mozilla 1.7rc3.. probably not the need to
stip the * out, but more the problem where if you have "10." in the no proxy for
list, and you go to http://foo/  where foo resolves to 10.x.x.x, it thinks foo
is outside the firewall and tries to use the proxy.
very frustrating here at work!

Comment 20

[Darin Fisher]

Comment on attachment 11158 [details] [diff] [review]
Patch for this bug

resolving hostnames here on the main UI thread is not ideal.  if DNS is slow it
could hang the entire browser UI for lengthy periods of time.  i would prefer
to see this happen lazily on a background thread.  but that is a very complex
patch to implement.

Comment 21

[nick]

but the dns name entered into the address bar must be resolved anyway if you 
are to attempt to connect to that site.. right? so once thats resolved, you 
can compare it against the no proxy for list.

Comment 22

[Christian :Biesinger (don't email me, ping me on IRC)]

(In reply to comment #21)
> but the dns name entered into the address bar must be resolved anyway if you 
> are to attempt to connect to that site.. right?

wrong: the proxy resolves the name, not the browser.
(except maybe in the case of a socks proxy)

Comment 23

[nick]

ok, but i guess this isn't really the point. IE does it, so it must be 
feasible..  this bug was brought up a long time ago (almost 4 years), should i 
start a new bug on it?  the fact that it hasn't been fixed in 4 years would 
suggest its not being looked at..

Comment 24

[timeless]

what makes you think that filing a new bug will get it looked at?
you'd be lucky if someone looks at it and recognizes it's a duplicate of this one.

anyway, the resolution we normally do (when we do some) for addresses doesn't
happen on the main thread, so at least from an abstract perspective, it would
seem to me that we should be able to do this work at the same time/place as
normal resolution.

nick: if you want to do something useful, how about building up a list of
situations and browsers and how they react.

the first one that comes to mind is:
ie, socks proxy, exclude: *.localdomain, http://foo/

local name server says foo=foo.localdomain
socks proxy says foo=foo.com

what does ie do?

be sure to specify os and ie version.

Comment 25

[nick]

sure, i'd love to help.

in the situation you described, entering http://foo/ into the address bar did 
NOT work.  entering http://foo.localdomain.com/ into the address bar DID work
OS: winXP sp1
ie vs: 6.0.2800.1106.xpsp2.030422-1633

same results for windows 2000 sp3, ie 5.50.4134.0600cO

however, in the proxy exclude list, if i enter foo, rather than http://foo/, 
then it works if entering http://foo/ in the address bar.  in this situation, 
entering http://foo.localdomain.com/ in the address bar still works.

further, if i remove all entries from the exclude list, so it is empty, 
entering http://foo/ into the address bar still works! also, if i enter 
http://10.16.17.88 (which is the ip of foo) into the address bar it still 
works.  however, if i untick "bypass proxy server for local addresses" in the 
LAN settings window, it will not work (only for http://foo/, 
http://10.16.17.88/ still works fine).  so it would seem that IE is somehow 
figuring out what's local and what's not without us entering anything into the 
exclude list.  

i guess im stuck as to what else i can test for you.  if you can suggest 
something, i'd be happy to try it

Comment 26

[timeless]

well, the definition of local could be 10.*, 192.168.*, 172.<i can't remember).

Try building a network using something that isn't in those ranges. (don't
connect it to the real world).

Comment 27

[Darin Fisher]

Comment on attachment 11158 [details] [diff] [review]
Patch for this bug

sr-

we must not block the UI thread waiting for slow DNS servers to respond.  we
currently do so for PAC, and that is unfortunate.  we should not add more code
that does this.

it is far better to use nsIDNSService::asyncResolve (which is what the browser
uses when loading non-proxy sites), but that requires that your code is happy
to get the response back asynchronously.  i don't think this code is, so we
need to drastically change the way things work.

christian and i spoke about a solution for PAC, and that is probably something
that we'd want to use here too.  it involves creating a dummy protocol handler
and channel that "redirects" to the proper channel after proxy settings have
been determined asynchronously.

i cannot accept this patch as is.  leave out the PR_GetHostByName for starters,
ok?

Comment 28

[Timothy Legge]

Hi All

Is this bug going to be dealt with in the near future?  I am trying to use
firefox in a corporate setting with a specified proxy.  The internal intranet
sites are listed in the no proxy for but the browser cannot find them.  If I
disable the proxy, it is able to find the sites without any issue.

Please address this bug as it is important for corporate users.

Tim

Comment 29

[benc]

Darin: 
the patch provided in this bug by justin is very cool. I like the idea that the
no proxy list would magically grow over time. However, I doubt we are ever going
to implement this, for several reasons:

1- there are limits to the size of a pref. This feature was not designed to be a
block list because it is saved as a pref. Other features that support domain
lists have heavy UI support (cookies, image blocking, etc.)

2- this is basically using code logic to magically failover in a situation that
could be solved w/o magic. This is an example of my growing believe that
network-magic features are bad. We can fix most problems in no proxy by fixing
our handling of the default domain at the pre-resolution stage. 

3- the "*" stripping was already implemented elsewhere, so it is irrelevant to
this bug now.

I've reverted the description back to something closer to the original. most of
the issues raised by nick are actually in other bugs, so I think we should
wont-fix this bug.

Comment 30

[Matt Blissett]

This may no longer be needed, as it's now possible to add <local> to the "No proxy for" list.

See https://developer.mozilla.org/en-US/docs/Proxies_in_Necko#Proxies_and_local_hosts

Comment 31

[Patrick McManus [:mcmanus]]

This isn't on anyone's work list and realistically is an abandoned idea. I will close as wontfix - if someone has a patch or is actively going to work on it please reopen. (but please, only then.)