150580 - Proxy: requests that result in a DNS error should cause domain-guessing to occur

Status: RESOLVED DUPLICATE of bug 2875

(Core :: Networking: HTTP, enhancement, P5)

Description

[Joel Thornton]

When an HTTP Proxy is configured in Preferences, instead of resolving text entered in the Location bar via DNS, the string is passed to the Internet Keywords parser, even when use of Internet Keywords is disabled.

Steps to repeat:

1. Set up a local proxy (in Windows, Proxomitron is suitable)
2. Make sure "Direct connection to the Internet" is set in Preferences->Proxies.
3. Type 'spam' in the Location bar; observe the DNS name is resolved normally.
4. Now in Prefs->Proxies set 'Manual proxy configuration' and enter the local proxy server details for HTTP Proxy (in the case of Proxomitron, the values are 127.0.0.1 and port 8080).
5. Repeat step 3. Observe that instead of resolving the domain name as expected, Internet Keywords lookup is done on 'spam'.

This is probably related to bug 127396 and bug 134105 (though neither make mention of the erroneous use of Internet Keywords instead of proper DNS resolution).

Comment 1

[Joel Thornton]

It is also worth noting that when this use of Internet Keywords happens, the
user_pref "keyword.URL", if set in the user's prefs.js, is not used, but always
defaults to using the search.netscape.com engine.

Comment 2

[Peter 'vt100' Schwindt]

confirmed using Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.1a)
Gecko/20020609 [2002060908]

* On typing 'spam' with no proxy configured --> getting to www.spam.com

* having a proxy configured --> Mozilla wants to connect to host spam.mydomain

Comment 3

[Joel Thornton]

Discovered the use of Netscape Keywords when using proxy was actually attributed
to my proxy server, NOT Mozilla!

DNS resolution still broken in Mozilla, but report on "Internet Keywords used instead" is erroneous.

Comment 4

[Peter 'vt100' Schwindt]

Removed the "Internet Keywords" part from the summary...

added "qawanted"

Comment 5

[Boris 'pi' Piwinger]

Is this still a problem in 1.1 or later?

pi

Comment 6

[Joel Thornton]

Yes, this problem still exists as of the 2002090608 nightly.

Comment 7

[Boris 'pi' Piwinger]

Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.2a) Gecko/20020906

I could verify your description. I think it is not a bug, though.

Case 1: Direct acess. Mozilla has to resolve the hostname to connect to the host.

Case 2: Proxy access. Mozilla only needs to talk to the proxy. The proxy will
take care of the rest, including resolution of "spam" in your example.

Do you agree? If so, please mark bug as INVALID. Else please exlpain.

pi

Comment 8

[Joel Thornton]

Taken from a purely usability standpoint, the problem is that typing "google" 
into tha address bar when proxy is in use does not put you at 
"http://www.google.com" because the DNS lookup is not done in the same manner as 
when proxy is disabled.

I think when proxy is in use, Mozilla should still resolve DNS names in the 
conventional manner.  This mirrors the behavior of IE and Opera under Windows; 
they both succeed in DNS resolution when a proxy is being used.

Many proxies being used by Mozilla users do not handle DNS resolution on their 
own -- they are intended purely for proxying HTTP, HTTPS, or another protocol.  
Ad-blocking proxies and anonymizing proxies are the most common examples of 
these.  If Mozilla wants to use a proxy for DNS resolution, then it should really
have a DNS Proxy preference for that specific purpose.  To proxy HTTP should not
be to proxy DNS as well.

In summary, if one wishes to use a proxy for HTTP/S under Mozilla, then DNS 
resolution will most likely be broken and unuseable by them.  This is why I think
this bug is still valid.

Comment 9

[Darin Fisher]

it would not be too difficult to add support for this.  we can easily look for
the proxy server's error code, and then trigger the usual URI fixup that occurs
when a normal DNS failure occurs.

Comment 10

[benc]

Joel, a couple comments:

#1
keyword.URL works, you should confirm that your changes are loaded from prefs.js
by using the about:config display.
#3
I think I understood #3, but not completely. Perhaps you could email me offline,
and if something for the bug needs to be clarified, put the conclusion here.

#8 
I you are really saying: I want all the URLfixup features of your browser in
your proxy. This is probably hard to do, because of the variety of vendors.
Netscape's Proxy seems to have adopted a couple bits, for example, the proxy
server does "Domain Guessing" for you...

GET http://packetgram/ HTTP/1.0

HTTP/1.0 302 Found
Proxy-agent: Netscape-Proxy/3.52
Date: Sat, 07 Sep 2002 06:08:43 GMT
Location: http://www.packetgram.com/
Content-type: text/html
Content-length: 212

I think this is a different problem than delegating the DNS of the HTTP request
to the Proxy. This model is the way HTTP proxy was designed, and it makes sense
for many reasons, especially centralization of network resources. 

What you are really asking for, it seems is that the larger process of what I
call "URL resolution", the taking of a user entered string, and doing domain
guessing, IK, search engines, etc to work transparently through these proxies.

In retrospect, Ari and the rest of the HTTP 0.9 -> HTTP 1.x writers should have
abstracted all the client networking errors so that a proxy (which is basically
webserver hooked into an http client) could receive could be cleanly sent back
to the original client in an unabiguous manner.

Many servers send back some kind of 500 class error, which is useful, but not
sufficent to help the client decide what to do next, say an Internet Keyword or
guess of the domain.

I think, ultiimately, you should look at bug 2875, which this might be a
duplicate of...

darin: there are comments in that bug as well about what HTTP status codes are
returned by some servers, which might help w/ what you are suggesting.

Comment 11

[Joel Thornton]

What I'm actually after here is to have all the URL resolution features *not*
worked out by proxy; if a proxy is in use, continue doing DNS resolution
in the standard, non-proxied manner.  I'm not aiming for getting DNS resolution
handled by proxy in any manner, since the proxies I use aren't designed for
that task.

Some folks may prefer to have DNS resolution be handled by their proxy; so
perhaps adding a DNS Proxy setting or a "Use proxy for DNS resolution" toggle
in preferences would meet this need.  I'm simply observing that IE, Opera, and
other browsers I've used in the Windows environment don't use the proxy for DNS
resolution when one is enabled.  They only use the proxy for network requests
to the corresponding service/port itself.

I hope this makes sense.  I'm not too clear on how DNS resolution works in
general, I just find that resolving DNS through the HTTP proxy is not the
behavior I want, as URL resolution simply does not work in this scenario; and
proper URL resolution with an HTTP proxy is all I'm after.

Please respond if you need further clarification.

I don't think this is a dupe of bug 2875, but it may be a dupe of bug 127396,
and bug 134105 sounds similar but appears to be different in the specifics.
:)

Comment 12

[Boris 'pi' Piwinger]

I think, this really is a dupe of bug 127396. But since Darin took over this
bug, I will leave this for him to dupe it one way or the other.

I tried to find some RfC which talks about that issue, but could not find it.
Maybe someone can point to source.

Bug 134105 seems to be exactly the opposite request, bug for SOCKS5. I don't
know if this can be dealt with in one fix (if any) or has to be separate.

pi

Comment 13

[benc]

Pi is right about the SOCKS bug. In SOCKS, we do DNS resolution locally, then
ask the SOCKS server to connect us to a socket (ip:port). SOCKS V5 added support
for delegation of DNS resolution.

Joel: Based on your last comment, what you want is a dupe of bug 2875. That bug
is in URL bar, and someone there should fix that summary. The reason it doesn't
sound like what you want is because the summary is inaccurate and the discussion
was side tracked for a while. I've changed your summary to be more accurate for
what you are asking for.

I looked at IE briefly, and it does NOT do DNS resolution locally for proxy
requests. If this is working w/ some browser+proxy combinations, making the
change you suggested would probably not "fix" mozilla. What we need is proxy
logs and maybe some packet traces to see what its going on.

What we really should do is revert your bug back to the original problem, and
focus on explaining what is going on.

Comment 14

[Darin Fisher]

not sure if i'm actually going to have time to get to this by moz 1.2

Comment 15

[Darin Fisher]

not going to happen for a while :(

Comment 16

[benc]

Okay, lets get this settled now:

Is this bug about 

1- local DNS of HTTP proxy requests?

2- improving HTTP proxy error handling to recognize proxy-side DNS failures?

I think it should strictly be about #1, because now that I think about it more,
bug 2875 is #2, and it affects all HTTP (URL) proxy requests, not just IK, but
also domain guessing.

I'm concerned about the viability of this change, without sitting down and
mapping out how this affects the relationship between several complicated
events: user-string entry, local browser fixup to URL, possible DNS resolution,
possible browser prefs resolution (PAC or "no proxy), Proxy request, proxy
response parsing. Most of these systems were designed wihout thinking about the
total process we are studying here.

The lack of robust IPv6 handling in our proxy subsystems in itself a huge problem.

I'm not saying we shouldn't do this, but Darin, could we sit down in front of a
really big marker board and think about any possible dependencies before you do
this?

Comment 17

[Michael T. Babcock]

Can I point out that you can get around this bug entirely by using the auto
proxy URL instead (and making your own local auto-proxy file).

This works well (change 192.168.1.1:3128 to your proxy server's address):

function FindProxyForURL(url, host)
{
        // Catches non FQDN addresses (keywords):
        if (isPlainHostName(host))
                return "DIRECT";

        // Catches unresolvable names (treats as keywords for searching):
        if (!isResolvable(host))
                return "DIRECT";

        return "192.168.1.1:3128";
}

Save that function in a file (like autoproxy.js) and use it.  I have more
complex versions as well, and other examples can be found on netscape.com.

Comment 18

[Allan]

Comment #17 does *not* solve this bug in any fashion. The auto-config for
proxies doesn't allow the changing of the entered URL, nor can it be used to
force Mozilla to do a DNS lookup on it. All it allows is for various proxies to
be used depending on testing of the entry in the location bar.

If, as in my case, the proxy must be used for all cases, but doesn't handle
"hotmail" as "www.hotmail.com" (but does under IE), this won't solve that.

Comment 19

[benc]

If you want redirection of your URLs, that can pretty much be only implemented
on your proxy. I don't think end users + security people are ready for a feature
like that.

Comment 20

[Allan]

Perhaps I'm not making myself clear. I don't think I want redirection. What I
want to do is type "google" into the location bar, and be taken to
www.google.com -- or type X to be taken to www.X.com. This works in Mozilla if
you have a direct connection, but if you use a proxy server (my workplace uses
Squid) "X" gets passed directly to it, and naturally fails.

However, Internet Explorer handles this perfectly, and automatically passes
www.X.com.

Comment 21

[benc]

Allan: That is bug 2875. To many people, that is not what it sounds like, but it
is, trust me.

In fact, this entire bug is a dupe of that bug, but the summary has never,
despite attempts to improve it, described what you really want.

These bugs always get dragged out and confused b/c we don't have a single design
document so this can make sense to people. I've changed the summary of 2875, so
it makes more sense what is needed to get IK (the original complaint in that
bug) and Domain Guessing (the original complaint here) to work when there is
something like a proxy-side DNS error from the first request.




*** This bug has been marked as a duplicate of 2875 ***