Last Comment Bug 2875 - Proxy: map HTTP 500 errors to necko errors (so Internet Keywords and Domain Guessing would work)
: Proxy: map HTTP 500 errors to necko errors (so Internet Keywords and Domain G...
Status: NEW
Product: Core
Classification: Components
Component: Networking: HTTP (show other bugs)
: Trunk
: All Other
P5 enhancement with 13 votes (vote)
: Future
Assigned To: Nobody; OK to take it and work on it
: Patrick McManus [:mcmanus]
: 150580 269519 336734 472127 718596 798856 (view as bug list)
Depends on: 20400 88217 127872
Blocks: 10276 104166
  Show dependency treegraph
Reported: 1998-04-23 07:00 PDT by Erik van der Poel
Modified: 2016-01-19 10:44 PST (History)
32 users (show)
See Also:
Crash Signature:
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Description User image Erik van der Poel 1998-04-23 07:00:00 PDT
(This bug imported from BugSplat, Netscape's internal bugsystem.  It
was known there as bug #114981
Imported into Bugzilla on 02/03/99 18:48)

If you set an HTTP proxy (e.g. and type an
Internet Keyword in the Location field, the proxy comes back with an error
saying that it could not find the host (using the keyword as a hostname).

When you type in "dogs and cats" it will work.

What's happening is that in the proxy case its using
the proxy before adding the "go " to the front.
Comment 1 User image Erik van der Poel 1998-04-23 22:50:59 PDT
I have a slightly different theory about what's happening. When you set an
HTTP proxy, the client uses it to try to resolve the hostname (or what it
thinks must be a hostname, even though we've typed a keyword).

If you type "toyota", the proxy turns that into "", and succeeds
in finding it.

If you type "blurfl", the proxy fails to find a host called "blurfl".
Comment 2 User image Erik van der Poel 1998-05-04 15:54:59 PDT
Guha will be out of town soon. The 4.x branch will be merged into Nova
on 5/11/98. Does this bug need to be fixed before the Nova merge?
Comment 3 User image John Giannandrea 1998-05-14 15:08:59 PDT
I wasnt aware that any proxies did the "" trick.  Most dont.

Anyway, the problem is that there may be no DNS on a client using proxy and
so the algorithm needs to be modified.  Since some proxies do indeed
resolve unqualified names (e.g. warp) then when someone types in a single
keyword then netlib must give it to the proxy for resolution.  This
implies that netlib needs to know why the proxy failed.  Since the
proxy doesnt distinguish between failed and failed DNS in the result
code (I think) then netlib is going to have to treat 404 and other error
codes differently in that case.  Refering to Judson for an opinion...
Comment 4 User image Erik van der Poel 1998-05-14 15:29:59 PDT
Date: Fri, 24 Apr 1998 00:22:33 -0700

Erik van der Poel wrote:
> Hi Lou and Ari,
> If Navigator is using an HTTP proxy, and it asks the proxy to resolve
> a URL that contains a non-existent hostname (i.e. gethostbyname()
> fails in the proxy), what is the proxy supposed to return? Is it
> supposed to return 404, or 400, or what?

Our UNIX proxy returns 500.  It shouldn't be 404 because I think HTTP
status codes refer to objects within servers, not the existence or
non-existence of servers themselves.  Since the proxy is not the
originator of the data, it cannot definitively determine the existence
or non-existence of a server (or an object within a server) -- in a case
where it fails to connect to one (regardless of whether the reason is
DNS, server down, or midstream proxy down).  Only an origin server can
say something like "404 Not found" or "410 Gone".

It shouldn't be 400 because the request format itself was ok.  I further
think that it shouldn't be any 4xx class status code (client's request
bad), since the problem may be a temporary DNS problem, or a problem on
the proxy itself.  I would therefore continue to vote that 500 (or some
other 5xx class status code) is the correct code for this case.

Our NT proxy may return some other status, though.  I don't have one up
right now so I can't check.  I'm Cc'ing Neel who might be able to answer
that.  --Neel?  --Thanks!!

Ari Luotonen, Mail-Stop MV-068          Opinions my own, not Netscape's.
Netscape Communications Corp. 
501 East Middlefield Road     
Mountain View, CA 94043, USA            Netscape Proxy Server Development


Neel later replied that the NT proxy also returns 500 in this case. -- erik
Comment 5 User image Erik van der Poel 1998-05-14 15:32:59 PDT
Note also that Ari sent my email to Jim Gettys (HTTP guy) in order to get the
HTTP spec itself tightened up in this regard (i.e. lack of specification for
this case).
Comment 6 User image Judson Valeski 1998-05-14 15:53:59 PDT
note: what we've done has completely removed the client's ability to explicity
notify the user that a host can't be found. Anytime a dns lookup fails (local or
proxy), the text that failed is sent to

someone needs to resolve what the http response code from the proxy server is
that we need to catch when the proxy's dns failure occurs. Once we have this the
"send off to" code can be injected where we catch http
response codes.

This problem is magnified when Proxy Auto configuration is used.
Comment 7 User image Judson Valeski 1998-05-18 15:04:59 PDT
reassigning to guha as he's apparently the one who's working on keyword stuff.

I just spoke with Ari and he said Erik has the proxy return code info. All that
needs to happen here is the proxy return code needs to be caught in
mhttp.c>net_parse_first_http_line() and the keyword call(s) need to be inserted
(this should be similair to the 304 redirect case I'm thinking).
Comment 8 User image Erik van der Poel 1998-05-18 15:46:59 PDT
Yes, I pasted the proxy return code info into this bug report. See above.
Comment 9 User image shani 1998-06-04 10:42:59 PDT
removed Rick Takeda as QA assigned - no longer with NSCP.
Comment 10 User image rubydoo123 1998-06-04 16:09:59 PDT
assigning Greg as QA Assigned to, please assign tothe appropriate engineer
Comment 11 User image glynn 1998-06-04 18:17:59 PDT
Ok, bear with me as I am not a proxy guru. 6/2 Win32 4.06 build.
 I manually entered "" and configed for manual proxy,
tried "toyota" which resolved and then tried "dogs and cats" and "blurfl".  The
latter two returned:

 "Error, The requested item could not be loaded by this proxy.  Netscape Proxy
is unable to locate the server: The server does not have a
DNS entry.  Check the server name in the Location (URL) and try again"

I take it this is what should occur, and this bug is fixed, or should it have
resolved and dumped me to the search engine?
Comment 12 User image Erik van der Poel 1998-06-04 20:05:59 PDT is currently mapped to uruk, an internal machine (inside
the firewall). is outside the firewall, and cannot find in the DNS database. will properly
exist outside the firewall when it is ready.

In the meantime, it would be best to verify this bug fix by using an internal
proxy (inside the firewall). I have one running at bluejay:8080. Please try
this. Essentially, you should get the same results, whether or not you are
using a proxy. If not, this bug has not been fixed properly.
Comment 13 User image glynn 1998-06-08 11:08:59 PDT
Ok, used bluejay as proxy vs. direct connection.  Enter "toyota", resolved
properly to toyota site.  Enter "mice and men" dumped to search engine in both
cases.  Enter garbage URL and with proxy I get the Error message; on direct
connect I get dumped to search engine.  So for this last test it would seem we
are still broken since the internal proxy should have worked to access uruk
the internal server?  Or am I misunderstanding the last explanation?
Comment 14 User image Erik van der Poel 1998-06-08 11:36:59 PDT
I think you performed the test correctly.
Comment 15 User image glynn 1998-06-08 20:44:59 PDT
Reopening as we apparently are still broken
Comment 16 User image guha 1998-06-09 19:16:59 PDT
I have no idea how to fix this or whether this is even a bug.
Yes, if some proxies behave badly, keywords won't work and
the browser will work exactly as 4.05 used to work. But there
is nothing we can do about it.
Comment 17 User image glynn 1998-06-15 16:50:59 PDT
Assigning to erik for re-evaluation....please see Guha's comments
Comment 18 User image Erik van der Poel 1998-06-15 18:07:59 PDT
If you enter a "garbage URL" (with non-existent hostname), then the proxy will
fail to lookup the host in DNS, and return an HTTP status code of 500. There
may be some way to detect this in the client, and then to failover to the
search query instead. I am not that familiar with libnet. One of the libnet
experts would be able to comment on the feasibility of such a fix.
Comment 19 User image chris hofmann 1998-07-16 16:44:59 PDT
so if we don't ship SB features this goes away.
if we do its a release note item.
Comment 20 User image glynn 1998-07-16 19:58:59 PDT
Clearing fixed rez as it is not fixed yet, and it appears from last entries this
needed some more look over and if SB stays in a possible fix.
Comment 21 User image chris hofmann 1998-07-20 08:33:59 PDT
assign to jud for one last look, then going to close it as wont fix.
Comment 22 User image leger 1998-08-07 19:53:59 PDT
Is pondering done?  would like to mark as Won't Fix and Release Note.
Comment 23 User image leger 1998-08-09 18:12:59 PDT
Putting in Status Summary comments.
Comment 24 User image Marek Z. Jeziorek 1998-08-11 14:33:59 PDT
Changing TFV to M14. valeski, could you either confirm that it will or will not
be fixed in Nova?
Comment 25 User image Jean-Charles Mourey 1998-08-25 19:42:59 PDT
Judson, we need a determination from you on whether this will be fixed in 4.5 or
whether we'll declare that certain proxy servers won't work with Smart Browsing.
 I believe you're in the best position to make that call from a policy
and standard standpoint.
Comment 26 User image glynn 1998-08-26 17:18:59 PDT
Last entry is from mourey asking for final input yet bug is marked won't
fix? reopening to flag for final input from Judson.
Comment 27 User image Judson Valeski 1998-08-26 17:35:59 PDT
I've indicated what would need to be done, whoever is familiar with the keyword
code (Guha I think) can take my suggestion and run with it, otherwise we'll just
need to take Chris Hofmann's suggestion to release note that keywords don't work
through proxies. I'm reassigning to guha to get this off my plate.
Comment 28 User image Peter Trudelle 1998-08-27 19:08:59 PDT
resolving as wontfix per bug triage mtg.
Comment 29 User image kenh 1998-08-27 19:24:59 PDT

I understand that our plates are too full, but please consider that Internet
Keywords is considered one of the most strategic things we as a company are
doing. I really think that we should fix this.
Comment 30 User image leger 1998-08-28 14:53:59 PDT
jar - can you help with this one?  Moving to 4.5b2 whilst more discussion

kristif - this is a Release Note for M14.
Comment 31 User image Jim Roskind 1998-08-28 19:22:59 PDT
I looked at this bug, and I'm pretty sure it is a proxy server problem.
I used bluejay in my test, per instructions above, except I used telnet to
manually test the proxy (I had to guess how to use it), and I was able to see
the actual return codes.

The first time I tried:
telnet bluejay 8080
GET http://dogs and cats

The response I got began:
HTTP/1.0 500 Error from proxy

That response caused the keywords function to work like a charm (when I tried
this with Navigator 4.5).
Alas, when I tried the following:
telnet bluejay 8080
GET http://nevernevernevernever

Then I did *NOT* get the leading error code!  All I got was the error page.  Try
it yourself using telnet.  Perhaps I don't know how proxys work (very possible),
but the above test seemed to show that bluejay did *NOT* return a leading error
code, or any of the common header stuff (re: mime-version, proxy-agent,
content-type) that should appear before the page.  As a result, the navigator
can't detect the fact that a page was NOT served.
Simply put, this is a server error (unless someone can tell me how a proxy
server is supposed to work.).

We could try to work around this proxy server error, but it would be hard
(actually impossible to be perfect... and a hack all the way).  I think the
correct resolution is a Wontfix, but it should really be raised as a bug to the
proxy server folks.

For now, we should release not the fact that a solo keyword will not put you
into the search system if a Netscape proxy is used.

My apology in advance if I'm misunderstanding how to mess with a proxy server...
but my tests sure explain the behavior we're seeing on Navigator.
Comment 32 User image luotonen 1998-08-28 22:15:59 PDT
No, you wrote an invalid request to the proxy.
First of all, an HTTP/1.x request looks like this:

	GET http://hostname HTTP/1.0
	Header: value
	Header: value

So you got the "HTTP/1.0" or "HTTP/1.1" missing from after the URL.

Secondly, it's illegal to type in spaces into the URL when it's in
HTTP.  You gotta encode them with  .  So "dogs and cats" should
issue the request:

	http://dogs and cats

Third, spaces are illegal characters in DNS hostnames, so if the URL
has spaces in the hostname portion, the Navigator could immediately
determine that it's a keyword search.
Comment 33 User image Jim Roskind 1998-08-28 23:29:59 PDT
Interesting.  I had tested my conjecture about how to contact a proxy server
with the telnet sequence:
GET http://florida/index.html

and had retrieved my web page (with a success code!). I was lulled into a sense
of understanding by that premature success.

Ari's correction to my attempt at a protocol suggests I was exploiting
"undefined behavior" in the proxy (getting responses without specifying the
HTTP/1.x protocol).

The interesting question is then what exactly is 4.5 sending to the proxy?  If
it is sending the correct request format, then it is an error (bug) that the
Navigator does not properly parse the Error Code (and toss the URL into the
Keywords handling code).  This might also fit with Ari's other comment, that the
Navigator could (and since this works, probably does) realize that spaces in the
 wanna-be-host-name preclude it from being a valid URL, and cause the keyword
code to handle the case without even contacting the proxy (and hence never
having to parse a proxy error code!).  Is there a log on a proxy server to
verify that the request is full and valid (including the HTTP/1.x element)?

I'd like to understand this a little better... as we'll eventually have to later
this to 5.0, since this bug is way beyond the current crash/data loss triage
level for 4.5.  This investigation (and set of comments) can at least be used on
a 5.0 fix.

Thanks in advance for all contributions,
Comment 34 User image luotonen 1998-08-31 19:04:59 PDT
Telnet'ting to the proxy as:


is the old, HTTP/0.9 protocol.  It is defined such that it does *not*
send a status code, just the document (or text form error message) back
to the client (for simplicity).  So in fact, the proxy is working ok
even in this case.  Note that the empty line is not present in this
format, either.

I've set up a proxy at:

You can look at the access and error logs at:


Let me know when you're done and I can shut this one down.
Comment 35 User image Jim Roskind 1998-08-31 20:53:59 PDT
Thanks to Ari's comments, I can now summarize the bug (correctly this time).
This is not a server bug (I was wrong), the problem is that several distinct
errors are lumped together by a proxy server, and we have to decide if we want
the Navigator to catch all such causes, and redirect handling to the keywords

The problem only appears when a solo word is supplied, and can be mistaken for a
host name.  The proxy server returns an identical return code (500) for the case
 of "Server does not have DNS entry" and "Server may be down or unreachable."
As pointed out by Valeski in detail on 5/18 above, folks *could* catch the 500
error and redirect it to the keywords handling code.  As he also pointed out,
this would mean that we would loose the ability to distinguish down-machines
from non-existant-machines when a proxy was active :-/.

Hence, if we "fixed" this bug, and always passed off 500 error codes to
keywords-handling, then we would have a new bug :-/.  We could then tell if we
are using a proxy by typing in a URL to a host that has a server down.  When a
proxy is not in use, then we'll be told that the host is down (via a pop-up).
When a proxy is in use, we'd be deprived of this info, and be fed off to

The new "alternate bug" would say "Internet Keywords don't work when machine is
down unless proxy is in use."   If not that, then it might say "Internet
Keywords are too aggressive when server is down and proxy is in use."  I
wouldn't know which to file... but there would be a clear (bug) difference
depending on whether you are using a proxy or not :-(.

IMO, the current bug is not that much "worse" than the bug that we'd get from a
fix.  A techie would like the current bug, which tells you when a machine is
down.  A non-techie would probably rather hear about other related sites when
his target machine is down (since a non-techie doesn't even follow the
distinction between client, server, net, and ISP).

IMO, without a clear spec (with some justification), this is a "bug" that should
not be fixed.  Folks could even decide to change proxy servers... and then we
could keep clients on identical footing with/without servers... but I don't
honestly know what the desired behaviour should be... perhaps even another
preference :-( :-(.

I'll now sit on this bug until some compelling arguments appear for
"the right" behavior (whatever that is)... or till the triage requirements are
enforced, and this non-crashing bug is latered (current triage rules state that
is our current 4.5 policy).
Comment 36 User image Jim Roskind 1998-08-31 23:36:59 PDT
I am latering this from 4.5 to 5.0 in response to further triage requests.  We
need to figure out what we want to do... and if we want to be consistent... and
then do it.
Comment 37 User image Jim Roskind 1998-09-03 20:12:59 PDT
This is not, IMO, a P0 bug.  It is not stopping folks from testing... nor
blocking other engineers from doing their work.  There is a feature here that is
 not spec'd completely... and that is really what is holding up a resolution.
See my comments above for a detailed explanation of the options, and
I'm downgrading this to P1.
Comment 38 User image Steve Dagley 1998-09-24 18:13:59 PDT
Not an FE bug since all the keyword magic happens in the backend, including the
redirect to  Reassigning to dp.
Comment 39 User image Suresh Duddi (gone) 1998-10-02 01:54:59 PDT
Jud could you keep this in your pool and decide based on jar's comment.
Comment 40 User image Paul MacQuiddy 1999-02-03 18:49:59 PST
Jud not around, giving to dp.
Comment 41 User image ebina 1999-03-11 12:16:59 PST
I'll accept just to get it off the NEW list, but I can't do
anything with it until the feature behavior is decided on.
Comment 42 User image pauld 1999-03-11 12:42:59 PST
removing myself from cc: list
Comment 43 User image glynn 1999-03-11 16:56:59 PST
qa contact set to
Comment 44 User image glynn 1999-03-11 16:56:59 PST
qa contact set to
Comment 45 User image kenh 1999-03-12 12:59:59 PST
JAR's analysis of the bug is close to correct, but he is failing to note that
the current behavior for single words w/o a proxy matches what he says would be
a bug if we fixed this bug. I.e., right now if you type an intranet server name,
and that server happens to be down, the 4.5 client will time-out and send the
single word to the keyword system. Judson Valeski was just suggesting that we do
the same thing for proxy server cases - if the proxy server returns a 500, then
the client should roll over to send the word to the keyword system. Since 99% of
the single words people are typing are likely to be internet host names, and not
intranet host names, these behaviors (with or w/o proxy) support the majority.
Comment 46 User image ebina 1999-03-12 16:04:59 PST
Ok, I understand.  To point is moot until we support proxies
in 5.0
Comment 47 User image leger 1999-03-13 16:39:59 PST
Setting to M5 Target Milestone to get fixed before PR1.
Comment 48 User image John Giannandrea 1999-03-14 17:47:59 PST
If I understand the proposed fix correctly, we want the client
to notice some class of proxy errors and then retry the request as a
keyword.  This assumes that one would never want the user to see those
proxy errors and their corresponding HTML pages.
This all assumes that you can tell DNS or connection errors apart from
other HTTP errors returned by the proxy.  This is non trivial since
various proxies return various things and there is some overlap in meaning
for some of the error codes.  I dont think you would want to roll over to
keywords for all 4?? and 5?? error codes for example.  And the HTTP spec
isnt much help since its suggestion (504?) is not widely implemented
(its only for HTTP 1.1 for starters).

The latest Squid (for example) seems to return 503.  The RFC claims that others
return 400 or 500.  There was some talk lst year about doing something evil
like parsing the descriptive HTML for hints that it was a DNS error but that
is fragile.

Im of the opinion that the correct place to fix this is in the proxy itself.
Already today if you ask for http://foo the proxy needs to choose how to
turn that into a FQDN.  Its not clear that the default need be the domain of
the proxy server's host.  Squid lets you configure what happens with the
append_domain configuration variable.  It does nothing by default.  If you
could rely on that (which you cant) then you could choose never to send
single word hosts to the proxy in the first place.

I have some patches to Squid that implement keywords.  If the hostname has
no dot and the DNS resolution fails, then it treats it as a keyword.  I will
submit the changes to the Squid maintainers if no-one objects.  Perhaps it
will catch on with other proxies, then again, perhaps not.  -jg
Comment 49 User image Erik van der Poel 1999-03-14 20:15:59 PST
I know of only one keyword server (ours), but if others are created in the
future, and if we want to allow users to set their keyword server in the prefs
file, then we want the client (not the proxy) to make the keyword request. So
this is one drawback if we get the proxy to do the keywords. However, I don't
think it's common to change the keyword server in prefs (and, indeed, maybe
Netcenter doesn't even want people to change it :-)
Comment 50 User image kenh 1999-03-19 09:59:59 PST
I like the idea of fixing proxy servers, but am doubtful that it will solve this
problem for the majority of users.

Here's a question: if the default mode of the browser was that non-URLs were
sent to the designated keyword server (without a local host lookup), would this
bypass the proxy problem?
Comment 51 User image Erik van der Poel 1999-03-19 10:41:59 PST
As far as I know, some people use proxies because they don't have a DNS server,
in which case they couldn't access the keyword server unless we hard-coded the
keyword server's IP address into the client. Also, I heard recently that some
ISPs force their customers to use HTTP proxies by disallowing access to port
80 (HTTP) on other hosts.
Comment 52 User image John Giannandrea 1999-03-23 23:11:59 PST
IE5 autodiscovers proxies.
Comment 53 User image chris hofmann 1999-04-26 14:51:59 PDT
something to keep on the necko radar...
will this ever be possible...
Comment 54 User image Judson Valeski 1999-04-26 15:42:59 PDT
once we resolve how we want to treat proxy response codes wrt dns failure, this
can be fixed. The component of this really should change though to some new
component that will be handling keywords.
Comment 55 User image leger 1999-07-20 12:14:59 PDT
Changing all Networking Library/Browser bugs to Networking-Core component for

Occasionally, Bugzilla will burp and cause Verified bugs to reopen when I do
this in a bulk change.  If this happens, I will fix. ;-)
Comment 56 User image Matthew Tuck [:CodeMachine] 1999-08-05 03:00:59 PDT
Isn't the dependency r'ship the wrong way round, ie this should depend on 10276?
Comment 57 User image leger 1999-12-13 16:37:59 PST
Bulk move of all Necko (to be deleted component) bugs to new Networking

Comment 58 User image Erik van der Poel 2000-01-07 14:38:59 PST
Date: Mon, 27 Dec 1999 15:51:01 -0800
From: "Tom Williams" <>

I found that this same problem occurs even on Netscape 4.7 using our Apache
 web server's proxy capability.  If I browse "cdnow", I get errors from the
indicating that "Get http://cdnow/" can't be handled by the proxy.  When I
specify, everything works fine.
Comment 59 User image Judson Valeski 2000-01-19 15:47:59 PST
making this bug dependent on the proxy bug.
Comment 60 User image leger 2000-02-01 17:33:43 PST
Putting on PDT- radar for beta1 - due to being the same behavior as 4.7.
Comment 61 User image Judson Valeski 2000-02-04 08:06:41 PST
I'm pushing the milestone on this out. Same behavior as 4.x as leger points out.
Comment 62 User image Judson Valeski 2000-03-29 08:21:06 PST
not getting to this for awhile. marking it as an enhancement because it's an
interoperability issue w/ existing proxy servers.
Comment 63 User image Warren Harris 2000-04-06 16:12:10 PDT
Moving to M17 (also beta2).
Comment 64 User image Paul MacQuiddy 2000-04-10 23:29:53 PDT
spam, changing qa contact from paulmac to on networking/RDF 
Comment 65 User image 2001-03-19 09:43:52 PST
Does anyone have an update that describes this behavior in Netscape 6?

Also, can someone comment if this is in Mozilla, or is this a commercial feature
now. If this a Netscape-only feature, perhaps it needs to be moved to bugscape?

Compared to other functional proxy problems, this seems like a low priority, so
I'm not likely to analyze this problem anytime soon.
Comment 66 User image benc 2001-05-11 17:26:38 PDT
qa to me.
Comment 67 User image benc 2002-02-26 09:05:51 PST
After reviewing a variety of Location, DNS and IK bugs, I think the solution is
that when we hit a 5xx class error, we should call IK and request again.

My reasoning is this:

If the proxy is having internal errors (bad config, low memory, no internet
connectivity), then a second request will probably return a 5xx error again anyhow.

If the 5xx error is related a request forwarding failure (server down, not in
DNS, whatever), then by failing over to a IK request, you can generate some
information for the user.

The other solution would be to implement bug 88217 or bug 127872. These would
decrease the situations where a server 500 error would be the final result
presented to the user.
Comment 68 User image Ben Bucksch (:BenB) 2002-02-26 11:31:14 PST
IMO, it would be a privacy problem to fallback to IK, if the desired server is
e.g. down. See bug 127872 for discussion about privacy differences between DNS
and IK.
Comment 69 User image benc 2002-08-16 00:31:05 PDT
-> URL bar (for Internet Keywords)

When IK==on, anything failing in DNS is sent to the IK server, so for any HTTP
Proxy setup, this might be an issue now.
Comment 70 User image Brant Gurganus 2002-10-13 11:15:59 PDT
[RFE] is deprecated in favor of severity: enhancement.  They have the same meaning.
Comment 71 User image benc 2002-12-09 15:14:09 PST
*** Bug 150580 has been marked as a duplicate of this bug. ***
Comment 72 User image benc 2002-12-09 15:15:51 PST
Focusing the summary on the actual change needed to fix the problem. This
underlies problems w/ getting IK and domain guessing to work correctly in proxy
Comment 73 User image Doug Turner (:dougt) 2003-04-08 09:18:31 PDT
old bug.  i think that this is now invalid.
Comment 74 User image Darin Fisher 2003-04-08 12:24:56 PDT
hmm... old bug yes, but seems valid to me.  should be a RFE instead of a normal
bug though.
Comment 75 User image benc 2003-04-08 15:07:42 PDT
The IK and Domain Guessing owners should decide if they even care about working
via proxies. If they don't, I would like to WONTFIX thse bugs.
Comment 76 User image Joel Thornton 2003-04-12 02:18:03 PDT
Well, I'm not the owner, but I for one would like to have this working -- at least domain guessing; IK is less important. 

The method that seems to be used in other browsers is that the proxy setting simply doesn't affect the domain name lookup routine.  And it seems to me that an HTTP proxy should only be used for HTTP requests, not domain name lookups.  
Comment 77 User image Ben Bucksch (:BenB) 2003-04-12 02:34:15 PDT
> it seems to me that an HTTP proxy should only be used for HTTP requests, not
> domain name lookups.  

Are you suggesting to try to make a DNS lookup from the client, ignoring the
proxy? That will fail here, and I don't know how (long delay or resolution
failure), because the client might not have access to any DNS server or the DNS
server might carry internal names only or the DNS server setting might be
intentionally misconfigured to avoid and reveal abuse.
Comment 78 User image Joel Thornton 2003-04-12 03:05:13 PDT
I see your point.  

Elsewhere I suggested adding an optional 'DNS proxy'; if unset, the client would
attempt DNS lookup/guessing as if there were no (HTTP) proxy. I think this would 
prove to be a cleaner and more useful fix than trying to intelligently "guess" 
appropriate behavior from the proxy response. Those who want proxied DNS can
simply specify it.

For the record, my proxy server [Proxomitron] returns '400 Host Not Found'. So I 
think it's safe to say that proxy servers cannot be relied upon to return a 
predictable response for domain guessing (based on reports on other proxy server
responses in above comments).
Comment 79 User image benc 2003-04-14 08:04:50 PDT
BenB's points are right on the money. That is why SOCKS4 was considered useless
in the real world, and had to be replaced by SOCKSV5, which allows HTTP (URL)
proxy style DNS delegation.

The reason newer proxies do not have server-level support for domain guessing,
is because the world has changed, and it doesn't make any sense anymore, we just
don't have a replacement for it yet.
Comment 80 User image Joel Thornton 2003-05-05 12:12:46 PDT
I suggested the addition of a DNS-only proxy preference as an effective solution
to this problem. HTTP proxy servers perhaps *should* handle domain guessing (or
uniformly return an error code), but the fact is, they don't -- with the
consequence being apparently "broken" domain guessing in Mozilla. 

I think adding a DNS proxy pref would roundly resolve this issue without getting 
proxy-vendor-specific or breaking existing functionality in Moz. Those who need 
DNS proxying could specify it, and those who want DNS handled by the local 
machine can leave it blank. This would also have the side benefit of permitting 
users to specify a different HTTP and DNS proxy.

I haven't heard any comments specifically against this suggestion, so I'm
wondering if it makes sense for me to start working on adding this feature.
I don't want to begin work, though, if there is little or no chance of it
being accepted into the build tree. 

Please comment.
Comment 81 User image benc 2003-05-06 22:01:00 PDT
DNS delegation has been mentioned in other bugs, but I think you should create
your own bug.

It would not solve all problems, because DG and IK are both hooked into
docshell, not just the DNS service. Proxy servers abstract all connectivity, the
design aspects are broader than just DNS.
Comment 82 User image :Gavin Sharp [email:] 2005-02-02 15:38:47 PST
*** Bug 280854 has been marked as a duplicate of this bug. ***
Comment 83 User image Matthias Versen [:Matti] 2005-06-11 05:54:33 PDT
*** Bug 269519 has been marked as a duplicate of this bug. ***
Comment 84 User image Adam Guthrie 2006-05-05 12:41:15 PDT
*** Bug 336734 has been marked as a duplicate of this bug. ***
Comment 85 User image Matthias Versen [:Matti] 2009-01-05 11:04:33 PST
*** Bug 472127 has been marked as a duplicate of this bug. ***
Comment 86 User image Ciprian Dorin Craciun 2009-01-12 09:41:23 PST
Now, I admit I haven't gotten to read through all the comments, and just skimmed through them, but from what I've seen, there is no clear solution that anyone agrees on because each proxy has a different behavior, and the HTTP specification does not require a specific behavior.

Thus I would propose a simple hack: from what I've noted, almost all sane proxy servers issue an 400 or 500 family status code, and also issue as response body a minimal HTML page that says the error code and a short description. Now we could observe that the page contains an H1 header line with the words "host not found".

So my proposed solution is very simple and efficient almost all cases: if the proxy responds with a 400 or 500 status code and inside the page we can find either the line (with a case insensitive regular expression) <title>.*host not found.*</title>, either <h1>.*host not found.*</h1>, then we could assume that the host does not exist.

Now I don't know how easy is to implement this, so please don't kill me :)
Comment 87 User image Ben Bucksch (:BenB) 2009-01-12 13:11:20 PST
Ciprian, and if the proxy is translated? :) That's one of the many reasons why programmers hate string parsing and invented error codes.
Comment 88 User image Ben Bucksch (:BenB) 2009-01-12 13:19:52 PST
Oh, and my proxy says "No such domain" (not "host"), of course in a <h2>, not <h1>, just to annoy us, and the <title> contains nothing of the sort, it just says "502 Bad Gateway" plus some nonsense, although it's not the gateway that's bad.


Personally, I ended up considering this bug a feature. I don't want Google to know what I looked for just because I misspelled the domain/hostname.
Comment 89 User image Ciprian Dorin Craciun 2009-01-13 01:29:16 PST
Indeed if the proxy is translated (which happens in how many cases?) there could be a problem. So would be the <h2> header instead of <h1>...

But we could parametrize the regular expression that should be searched for (for example a key named network.proxy.http_host_not_found_re), and thus everyone could customize according to their needs...

As I said in the beginning what I've proposed is an *very bad* hack, but it is doable.


> Personally, I ended up considering this bug a feature. I don't want Google
> to know what I looked for just because I misspelled the domain/hostname.

For this there is the setting keyword.enabled, and I intend to use the keyword feature as a shortcut for my personal wiki (through the setting keyword.URL).
Comment 90 User image El_Hoy 2010-06-24 06:17:57 PDT
I don't know if it help, but I'm from Argentina, I have a proxy in the university and the response is:

HTTP/1.0 503 Service Unavailable
Server: squid/3.0.STABLE18
Mime-Version: 1.0
Date: Thu, 24 Jun 2010 13:05:24 GMT
Content-Type: text/html
Content-Length: 2203
X-Squid-Error: ERR_DNS_FAIL 0
X-Cache: MISS from localhost
X-Cache-Lookup: MISS from localhost:8000
Via: 1.0 localhost (squid/3.0.STABLE18)
Proxy-Connection: close

the contents are in english. Thanks for all.
Comment 91 User image Matthias Versen [:Matti] 2013-01-16 11:59:11 PST
*** Bug 718596 has been marked as a duplicate of this bug. ***
Comment 92 User image Matthias Versen [:Matti] 2014-10-04 10:53:21 PDT
*** Bug 798856 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.