91587 - Proxy: "no proxy for" default domain filtering fails w/ non-FQDN (e.g., http://web/)

Status: NEW

(Core :: Networking: Proxy, defect, P5)

Description

[Chris Schanck]

From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.2+) Gecko/20010716
BuildID:    2001071608

I work at company foo.com. If I have manual proxy configuration turned on, I
cannot got to web.foo.com IF I ENTER IT as just web. Even though  'web' by
itself should default to web.foo.com, it get's aut-expaned to www.web.com by Moz. 

Now if I have no proxying turned on, it workds fine, with web becoming
web.foo.com per my default domain search setup.

It does not seem to matter what I use in the "No Proxy For" field; .foo.com does
not solve the problem. Directly entering http://web.foo.com does work, but since
the entire company intranet uses http://web/blah, I effectively cannot use Moz
as a day-to-day browser.

I believe the corporate proxy is Netscape. This behaviour occurs using beth
Windows 98, WIndows NT Linux and Solaris 2.8.

Reproducible: Always
Steps to Reproduce:
1.Set up a direct manual proxy (i.e, proxy, port 5000) for http
2.Set "No Proxy For" to .foo.com (Or not, won't matter)
3.Enter just the name of the machine you are browsing to without the domain
suffix, i.e. 'web' for 'web.foo.com' when 'foo.com' is your domain.



Actual Results:  Watch Moz somehow resolve 'web' to 'www.web.com' (Doing best
guess algorithm here? Skipping a check for domain suffix expansion because of
proxies enabled?)

Expected Results:  Resolve http://web to http://web.foo.com

This works fine without a proxy. But it works fine both ways under Netscape 4.77
and IE 5.5, so it needs to be fixed. AS I said it prevents me (rest of company)
from using Moz as a day-to-day browser solution.

Comment 1

[benc]

+makingtest - I was wondering where that feature went, I'll investigate.

Comment 2

[Bradley Baetz (:bbaetz)]

actually, I noticed this today, but I thought it was a proxy config error. I
didn't look into it further though. CONFIRMING, although it may turn out to be a
proxy error.

Comment 3

[Doug Turner (:dougt)]

reassign back when you have a test case.

Comment 4

[Bradley Baetz (:bbaetz)]

OK. (this url will only work internally, obviously...) I may look into this.

Comment 5

[Bradley Baetz (:bbaetz)]

actually, this is INVALID. According to the squid docs, you need to set the
append_domain config variable (assuming you're using squid), since its dns code
turns off that resolver options.

I presume the something similar works for other proxy servers.

I don't know why ns4 works for you - it fails on squid, with the same problem.
I'm still guessing that its a proxy server issue though - we don't look do
anything with the same when forwarding it to a proxy.

Comment 6

[Chris Schanck]

While it may in fact be true for you, under squid, at both my previous job and
this, using Netscape's commercial proxy, it DOS NOT work for Mozilla. However,
IE 5.0 (Windows NT), IE 5.5 (Windows 98), Netscape 4.73 (Windows NT), Netscape
4.77 (Windows NT), Netscape 4.77 (Windows 98), Netscape 4.77 (Solaris 2.8),
Opera 5.12 (Windows 98) Opera 5.0 (Linux) all resolve this correctly. 

Is there an explicit documentation (RFC?) that governs how name resolves are
done with proxies? Because "squid doesn't do it this way" is a lousy answer to
those of us stuck behind other Proxies ;-). 

Not to be a PITA, I just want to be able to use Moz, and at this point it is
unusable in my environment (2500+ employee company).

Comment 7

[Bradley Baetz (:bbaetz)]

Well, this is odd, since the proxy should be doing name resolution itsself, and
the fact that ns4 didn't work with squid without this config option means that
ns4 isn't doing it itsself.

Here are some things to try. In all these examples, you want to get to host foo,
which is really foo.example.com:

1. does nslookup foo work? (It should, because you said it works without a
proxy). Does it give the same results as nslookup foo.example.com?

2. I need some packet traces, including the contents of the packets, not just
the headers. I use ethereal, but it doesn't matter what you use as long as I can
open it. Try connecting to http://foo/some_generic_page.html and
http://foo.example.com/some_generic_page.html from both mozilla and something
else, from the same machine. Make sure to label which is which :)

Strip out any proxy-auth lines, if your proxy uses a password :)

actually, rereading the bug, is the problem that you need the no-proxy-for, and
the proxy can't actually get at foo.example.com? I didn't test that, and I have
no idea what ns4 does in that case.

Comment 8

[benc]

doug: I think the test case is in here, it just needs cleaning up.

bbaetz: We should have gathered more data, like the proxy logs, and I should
have emphasized that from the start.

The Netscape Proxy server, to a large degree is (confessionally) much more of
the Commerce/Fasttrack/Enterprise server + client code (probably circa Netscape
3.0) than anyone wants to admit. That means, it seems to have much of the URL
name resolution behavior of the client. For better or for worse...

So I typed in: http://slip/ in mozilla to my Netscape Proxy and the logs showed:

205.217.228.185 - test [24/Jul/2001:09:56:24 -0700] "GET http://slip/ HTTP/1.1"
302 206 - - - - 487 171 - - 1 DIRECT FIN - DO-NOT-CACHE
205.217.228.185 - test [24/Jul/2001:09:56:31 -0700] "GET http://www.slip.com/ima
ges/newsclip.gif HTTP/1.1" 200 1284 200 1284 - - 553 322 549 322 1 DIRECT FIN FI
N DO-NOT-CACHE

etc...

The 302 response to the hostnamed URL was suspicious, so I tested another domain
w/ a packet trace (too lazy to clear cache).

http://packetgram resulted in the following response:

www.packetgram.com -> h-205-217-228-185.netscape.com HTTPS R port=1149 HTTP/1.0
302 Found\r\n

           0: 4854 5450 2f31 2e30 2033 3032 2046 6f75    HTTP/1.0 302 Fou
          16: 6e64 0d0a 5072 6f78 792d 6167 656e 743a    nd..Proxy-agent:
          32: 204e 6574 7363 6170 652d 5072 6f78 792f     Netscape-Proxy/
          48: 332e 3532 0d0a 4461 7465 3a20 5475 652c    3.52..Date: Tue,
          64: 2032 3420 4a75 6c20 3230 3031 2031 363a     24 Jul 2001 16:
          80: 3539 3a35 3420 474d 540d 0a4c 6f63 6174    59:54 GMT..Locat
          96: 696f 6e3a 2068 7474 703a 2f2f 7777 772e    ion: http://www.
         112: 7061 636b 6574 6772 616d 2e63 6f6d 2f0d    packetgram.com/.
         128: 0a43 6f6e 7465 6e74 2d74 7970 653a 2074    .Content-type: t
         144: 6578 742f 6874 6d6c 0d0a 436f 6e74 656e    ext/html..Conten
         160: 742d 6c65 6e67 7468 3a20 3231 320d 0a0d    t-length: 212...
         176: 0a                                      

If the original hostname resolves via the default domain of the proxy
(proxy.packetgram.com exists, slip.packetgram.com and packetgram.packetgram.com
do not...) then the hostname request is resolved and the page is returned:

205.217.228.185 - test [24/Jul/2001:10:14:19 -0700] "GET http://proxy/ HTTP/1.1"
 200 252 200 252 - - 489 221 485 221 0 DIRECT FIN FIN DO-NOT-CACHE

So the "bug" is actually the proxy server, which retains that old "hostname" ->
www.<hostname>.com behavior that I will once again state emphatically that I
dislike. A triumph of expediency over software design. 

As for why the other versions work I do not know. The behavior is the same with
me for mozilla 0.9.2 and Communicator 4.

chris: I'll see if there is an proxy-level configuration change. What would be
really useful is some log entries from your Proxy of this problem.

Comment 9

[Bradley Baetz (:bbaetz)]

benc: right. But that doesn't explain why moz acts differently to ns4.

Comment 10

[benc]

It would help it it were reproducible for me. We'll need more customer data.

Comment 11

[Christopher Aillon (sabbatical, not receiving bugmail)]

*** Bug 111289 has been marked as a duplicate of this bug. ***

Comment 12

[Jonathan Brecher]

Some more fuel to the fire...

I'm *not* behind a proxy, but I am behind a firewall.  For me, http://scandium/ 
should resolve to http://scandium.cambridgesoft.com/, but for you it shouldn't 
(http://scandium.cambridgesoft.com shouldn't resolve for you either).

When I *first* launch Mozilla after a reboot (MacOS 9), I find that typing 
http://scandium/ into the location bar gets resolved incorrectly as 
http://www.scandium.com/.  However, if I launch Netscape 4.7 first, and type 
http://scandium/ into its location bar, it correctly resolves to 
http://scandium.cambridgesoft.com/.  Once I've done that, Mozilla is then able 
to resolve http://scandium/ correctly.

I don't know much anything about network protocols, but it seems like something 
needs to be initialized at the system level to get the local domain recognized?  
It seems that Netscape 4.7 knew how to do that initialization, but Mozilla 
doesn't...

I have the source code building on my machine to resolve some other issues.  If 
someone tells me what to look for, I'd be happy to provide whatever source-level 
information about this dns stuff that I can.

Comment 13

[Darin Fisher]

what happens if you just type "ping scandium" in a terminal?  the behavior you
describe seems to correspond to a DNS lookup failure.

Comment 14

[Jonathan Brecher]

Sorry, this is OS 9 -- no terminal window.  Do you have a preferred tool for 
doing a ping on OS 9?

Comment 15

[Jonathan Brecher]

OK, I found a ping tool for OS 9 -- IPNetMonitor.PPC.  You're right -- at the 
times when Mozilla fails to resolve scandium, the ping tool fails as well.  But 
after I try to access the full domain name, then the short name works in all 
places.  

Weird.  I'll throw this to our network admin.  I was hoping I might have had 
something useful for this bug, but maybe not in the end.  Sorry for the false 
alarm.

Comment 16

[[e-mail]]

I think Open Transport has some DNS caching, and here's the problem.

If you are on a really slow ISP, then the first request for
http://scandium.cambridgesoft.com/

might not work on the first try because your resolver receives the DNS response
(which can involve some delays as your nameserver asks other nameservers for the
information). However, since DNS does return the answer (but later), subsequent
attempts do work.

I have this problem with one of my ISP's, ironically, the one that provides the
fastest throughput has the slowest DNS servers. There are days where typing even
popular domains (www.news.com) fails on the first attempt...

We are still probably getting nowhere with the actual bug. I would argue this is
another reason why bug 88217 is a good idea, we would never have to deal with
this abiguity 

Comment 17

[Darin Fisher]

the problem here (if i understand correctly) is that we do not resolve "web" to
it's fully qualified domain name before checking it against the "no proxy for"
preference.

resolving this problem might be very tricky given the current networking
architecture, which makes mean uncertain of how quickly we could resolve this bug.

targeting mozilla 1.0.1 for now.

Comment 18

[benc]

darin's comments are  really on point, and I think there is another bug that
this is a dupe of another bug, but I can't find it.

I think we got lost arguing about what the proxy does with non-FQDN hostnames in
requests. I'm not convinced this is ever a good URL to proxy, b/c of the
ambiguity, which I keep saying should be fixed via bug 88217.

Otherwise, the other comments bradley and I have made show the hazzards you get
when you leave your hostname-fqdn resolution to the proxy administrator's whims
(or worse, the proxy coder's whims).

As for the MacOS item, lets make that a new bug, since OT caching complicates
the problem.

Comment 19

[Darin Fisher]

mass futuring of untargeted bugs

Comment 20

[Daniel Wang]

dupe of bug 95390?

Comment 21

[Darin Fisher]

dwx: i think the two bugs are slightly different.  this one has to do with
consulting the local machine's DNS search rules (which is very difficult to fix)
and the other one, bug 95390, is just about recognizing that http://foo is not
an internet keyword.

Comment 22

[benc]

I've had problems finding this bug in the past, and I think I'm the cause,
because I changed the summary to something hopelessly inaccurate.

I'm thinking this should say: 

Proxy: "no proxy" should match hostnames using default domain.

I think this is what the reporter wanted.

Comment 23

[Darin Fisher]

yeah, the old summary was pretty bogus.  this will require a DNS lookup for
hostnames that do not have a dot ('.') in them.  this kind of preliminary lookup
will be useful in solving a number of other bugs as well.

Comment 24

[Darin Fisher]

*** Bug 127396 has been marked as a duplicate of this bug. ***

Comment 25

[Gert-Jan Vons]

Is there any news on this issue? I'm currently using Mozilla 1.4alpha under XP,
and typing FQDNs in order to reach the web servers on our LAN is a bit of a pain.

Same thing for links that don't use an FQDN, I have to patch them manually while
surfing, making navigation on the company's internal websites a rather
frustrating experience.

Comment 26

[benc]

You should consider writing a pac file that does what you want, and saving it to
local disk and accessing it via a file URL.

Comment 27

[Darin Fisher]

*** Bug 232337 has been marked as a duplicate of this bug. ***

Comment 28

[Oscar Manuel Gómez Senovilla]

Why not add an option like "no proxy for local network"? That would
automatically detect unsuffixed ulrs, and maybe non-public networks.

I just leave the idea, for focusing on covering something more universal (which
covers most specific issues) instead of something more specific by hand (and
that with the universal rule could remove most headaches).

Comment 29

[benc]

The bug where we talk about doing it the Microsoft way sort of died because
nobody could clearly describe what the test cases would be to implement it w/
full compatability.

This bug would ideally be fixed by implementing bug 88217, but maybe it is time
to implement this feature purely on a DNS FQDN-parsing basis.

I've been working on some of these areas in PAC, so, I'll take a look at this soon.

Comment 30

[OstGote!]

As workaround use "warp" or "web" in "no proxy for" for internal addresses like
http://warp/blah or http://web/blah. I do the same, fortunately only a few hosts.

Comment 31

[OstGote!]

(In reply to comment #30)
> As workaround use "warp" or "web" in "no proxy for" for internal addresses like
> http://warp/blah or http://web/blah. 

Sorry, it is not so easy as I thought. If you put for example "info" in the no
proxy field you will get direct access not only for http://info/ but also for
all http://*.info/ web sites. Strange, I thought for the latter I have to put
".info" there.

Comment 32

[benc]

Yeah. That is the problem with this feature, it treats the block list as strings.

PAC does this as well, right now, but I'll be fixing that really soon, hopefully
in time for 1.8f, at the rate I'm going.

Comment 33

[OstGote!]

*** Bug 271511 has been marked as a duplicate of this bug. ***

Comment 34

[benc]

*** Bug 115948 has been marked as a duplicate of this bug. ***

Comment 35

[benc]

The most minimal solution would be to live with the "matching by string suffix"
logic and shoehorn the comparisons to work.

[ ] Do not proxy hostnames without domain name.

Then you add a simple, isPlainHostname-like check after doing the IPv4 address
check on the hostname. If the host is a hostname, then if this is checked ->
noproxy.

If we supported a default domain, I'd prefer to append the default domain and
send it to the existing comparison code, but nobody seems except me seems to
think that is a good enhancement.

Comment 36

[taomyn]

Is this behaviour ever going to be sorted?

As far as I'm concerned the OS should resolve the web address, so if I enter http://servername and servername would resolve as servername.mydomain.com then that's what should be used.

Having to add separate "no proxy" entries for each non-FQDN in our Intranet is a nonsense and halts any idea of migrating to Firefox for many of our clients.

Comment 37

[Jerry Baker]

I hope I'm not missing something, but I'm getting the feeling that some are thinking that appending a default domain to a host name will fix this bug. It will not.

At my office we have hundreds of machines addressed by single host name (e.g., colin, vjer, etc.). Appending the domain name to them and trying to query DNS will return an error because the FQDN is different than [host].foo.com.

Comment 38

[Wouter D'Haeseleer]

How is the status?
This is an issue from 2001...

We have the same problem and this is the show stopper for migrating +1000 users to firefox.

All our systems have logical names and it is not user friendly to also add the FQDN to this name for our users (like each other end users)

Or is there a workaround for this nasty thing?

Comment 39

[George Bell]

I've been struggling with this one for a while.. at home it worked fine to point to local machines without using the FQDN, but at work it caused me grief.  Finally came to realize it was due to the proxy..
A relatively simple work-around (at least for me, hopefully for others) was to simply add some an exception to my proxy.pac - instead of trying to add each host (which is a huge pain with several thousand names) I just added the network range for example:

function FindProxyForURL(url,host) {
  if (isInNet(host, "10.0.0.0",  "255.0.0.0") {
    return "DIRECT";
  }
  else {
    return "PROXY proxy.company.com:8080; PROXY proxy2.company.com:8080";
  }
}

That way all hosts in the 10.x.x.x net (or whatever net you need) would bypass the proxy.  Not sure if this can be done without a proxy.pac, though..

Comment 40

[George Bell]

silly me, you certainly can add network ranges to the "No Proxy For" field: something like: 10.0.0.0/8 (or 192.168.0.0/24, etc) seems to do the trick.

Comment 41

[benc]

Jerry: re #37, DNS isn't involved. "no proxy" does a comparison between the URL hostname and a list of strings.

The proposal isn't to start tacking on domain names to hostnames, just tack it on for the purposes of comparison.

Personally, I think we should implement the checkbox solution, you can do this w/ a smart pac file right away... (I'll try to get to this soon...)

Comment 42

[Toby Collier]

This will definitely prevent a corporate deployment to our 10,000 systems.

Comment 43

[Ulisses Penna]

     I would like to discuss this problem a little bit more because I can see some related bugs with this one. We already have deployed Firefox in our 90,000 workstations and I could tell you the problem reported is a little bit anoying.
     As far as I could understand, the problem is related with hostnames that are no FQDN - for example: http://foo <- that is the main problem (I am also facing it in my company).
     I would like to make an explanation first and, please, correct me if I am wrong.
     If I supply a URL like http://foo (not FQDN) instead of http://foo.mycompany.com (Fully Qualified Domain Name) Firefox *should* do the following:

a) asks the resolver for a expansion name (in order to get the FQDN -reported in bug #88217);
b) try to match for "No proxy for" *after* the expansion name (I am almost sure it does the match first instead of the expansion first! Bug #136789)
c) if the FQDN hostname *is* in the list "No proxy for" so get the connection to the server and do the job ...;
d) if the FQDN hostname *is not* in the list "No proxy for" so connect to the proxy and son on ...;

     Acting that way, Firefox would handle URL's like http://foo/my_index.html with no problem at all since it would first ask for a name expansion and then look for the exceptions in the proxy list.
     However, Firefox does not act like that. 
     Instead, Firefox, *does* the following:

a) try to match the hostname (not FQDN) for "No proxy for". Example: http://foo
b) since that hostname is not at the "No proxy for" but only the domain is at the "No proxy for" (i.e.: .mycompany.com) then Firefox try to ask the Proxy to connect to the "foo" host and, in all the related bugs reported, the proxy fails. The problem could *also* be solved configuring the proxy server. However, IMHO, I do not think it is a job for the proxy since the hostname "foo" is at my intranet company and so does not need to connect to the "foo" host via a proxy server.

     The only problem I can see about my suggestion is that when I want to connect to the "localhost". Following my suggestion, Firefox would try "http://localhost.mycompany.com" and, of course, it would fail.
     However for the case for explicity hostnames (I mean not FQDN hosts) Firefox should look at the "No proxy for" first looking for *hostnames* for matching. At a first glance this could look like a chicken-and-egg problem but it is not. Firefox should treat "No proxy for" hostnames separetely from dealing with domain names.
     In sumary - or let me explain better, my final sugestions are:

a) if a single hostname is provided (not FQDN) Firefox should look at the "No proxy for" looking for *hostnames matches only* (if the option for using proxy is enabled - of course). If Firefox found an entry, so connect to the host. If not, go to the (b) below;
b) if a single hostname was not found at the "No proxy for", Firefox should ask the resolver for a FQDN of the host;
c) now with the FQDN, Firefox should look - again - at the "No proxy for" for matching *domain names*. If it is in the "No proxy for", so connect to the host; if it is not, then connect to the proxy;
d) then, follow the flow as Firefox does today;

     And if the user supplies the FQDN, at first, then Firefox should work on (c) as it does today and follow the flow.
      Someone could suggest to put a list of hostnames in the PAC file in order to Firefox read the proxy configuration. However, it is not a good solution since it would demand frequent edits at the PAC file. The solution proposed here, IMHO, could provide a better behavior for Firefox.

P.S.: I am NOT talking about resolving names to IP addresses first and then look at the "No proxy for" looking for address matches. This feature is reported in other bugs where people is asking Firefox to honor the /etc/hosts file.

Comment 44

[Peter Dräger]

(In reply to comment #35)
> [ ] Do not proxy hostnames without domain name.

this checkbox is a simple and good solution. are there any plans to implement it?

kind regards
peter

Comment 45

[Charles Hoff]

I am very glad I found this thread. I have spent a number of hours trying to figure out why Firefox is passing on intranet sites that I thought should have been on the No Proxy list. 

I finally came to the conclusion that Firefox was not even trying to resolve hostnames before looking at the list. I agree with Ulisses Penna here with how this work. By all appearances, the other web browsers that I use all seem to resolve the name one way or the other and work fine at this location, but not Firefox. 

This workplace uses an offsite proxy to filter internet, however the intranet corporate webpages many times simply have hostnames. (i.e. "http://server2/TimeClockSelect.asp") I have the system set to bypass proxy for 10.100.* addresses, and if Firefox is set to "Use System Proxy Settings" everything is fine. If I try to use Firefox proxy settings natively, then "10.100.100.0/24" nor "*.milmont.local" works as expected. I have to literally have "server2" on the list or it doesn't work.

But I also have to have the other two on the list for the times a network service uses an alternative address. The "Internet Explorer and other browsers" approach requires one, simple entry. The Firefox approach requires several entries, all for the same server. With a domain as small as this one is, this can be lived with, (albeit grumpily,) but if I were trying to administer something much larger than this I'm pretty sure I'd have to abandon Firefox for general usage, as much as I really like it otherwise.

Comment 46

[Matt Blissett]

A very basic change would be to document the "<local>" setting in the GUI -- suggestion below.  A box to tick could help too, even if it just adds and removes <local> from the list.

The <local> setting is documented here: https://developer.mozilla.org/en-US/docs/Proxies_in_Necko#Proxies_and_local_hosts . The documentation is also required by #72444.  I've submitted a very small change to https://support.mozilla.org/en-US/kb/advanced-settings-browsing-network-updates-encryption (where the "Help" button takes me), which is awaiting approval.


diff -r 53376ef850fc browser/locales/en-US/chrome/browser/preferences/connection.dtd
--- a/browser/locales/en-US/chrome/browser/preferences/connection.dtd   Mon Jan 27 13:13:48 2014 +0100
+++ b/browser/locales/en-US/chrome/browser/preferences/connection.dtd   Mon Jan 27 18:11:34 2014 +0000
@@ -39,6 +39,6 @@
 <!ENTITY  SOCKSport.accesskey           "t">
 <!ENTITY  noproxy.label                 "No Proxy for:">
 <!ENTITY  noproxy.accesskey             "n">
-<!ENTITY  noproxyExplain.label          "Example: .mozilla.org, .net.nz, 192.168.1.0/24">
+<!ENTITY  noproxyExplain.label          "Example: .mozilla.org, .net.nz, 192.168.1.0/24, <local>">
 <!ENTITY  shareproxy.label              "Use this proxy server for all protocols">
 <!ENTITY  shareproxy.accesskey          "s">

Comment 47

[Matt Blissett]

Attachment: Add <local> to proxy settings GUI

Previous patch wasn't quite right, the < and > needed escaping.

Comment 48

[Firefox Bug Husbandry Bot]

Bulk change to priority: https://bugzilla.mozilla.org/show_bug.cgi?id=1399258

Comment 49

[BugBot [:suhaib / :marco/ :calixte]]

In the process of migrating remaining bugs to the new severity system, the severity for this bug cannot be automatically determined. Please retriage this bug using the new severity system.

Comment 50

[Randell Jesup [:jesup] (needinfo me)]

Mike - Any thoughts on this for current-day usage?

Comment 51

[Randell Jesup [:jesup] (needinfo me)]

Either the checkbox idea, or the more complete solution from comment 43

Comment 52

[Mike Kaply [:mkaply]]

Considering we've had no comments or duplicates in 10 years, I'm not sure it matters anymore...

Comment 53

[BMO Automation]

Moving bug to Core/Networking: Proxy.