Last Comment Bug 235853 - [PAC] Defer proxy resolution for HTTP and HTTPS PAC to avoid blocking main thread during DNS resolution
: [PAC] Defer proxy resolution for HTTP and HTTPS PAC to avoid blocking main th...
Status: RESOLVED DUPLICATE of bug 769764
: hang
Product: Core
Classification: Components
Component: Networking (show other bugs)
: Trunk
: All All
: P1 critical with 96 votes (vote)
: mozilla13
Assigned To: Steve Workman [:sworkman] (INACTIVE)
:
Mentors:
http://www.ebay.co.uk
: 306922 391682 425146 425183 450153 483544 489772 491253 539922 590247 652496 (view as bug list)
Depends on: 97605 282442 475069 479328 767005
Blocks: fx-enterprise 79893 208287 309582 356281 450153 468079 750445 769764
  Show dependency treegraph
 
Reported: 2004-02-27 10:47 PST by Stewart Jeacocke
Modified: 2015-07-29 05:36 PDT (History)
106 users (show)
benjamin: blocking1.9.1-
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
Improper connection settings UI (18.09 KB, image/png)
2007-06-06 08:58 PDT, Anthony Guselnikov
no flags Details
resolve non-blocking, and if we block defer until later (1.80 KB, patch)
2008-03-24 10:12 PDT, Mike Shaver (:shaver -- probably not reading bugmail closely)
no flags Details | Diff | Splinter Review
With fix to newProxyInfo, and restriction to http/https (3.17 KB, patch)
2008-03-24 15:01 PDT, Mike Shaver (:shaver -- probably not reading bugmail closely)
cbiesinger: review-
Details | Diff | Splinter Review
Better fix for protocol restriction (3.23 KB, patch)
2008-03-24 19:29 PDT, Mike Shaver (:shaver -- probably not reading bugmail closely)
cbiesinger: review+
Details | Diff | Splinter Review
posterity: from rediffing after removing the NS_ASSERTION, not just rebuilding and running :( (3.10 KB, patch)
2008-03-25 06:32 PDT, Mike Shaver (:shaver -- probably not reading bugmail closely)
mconnor: approval1.9b5+
Details | Diff | Splinter Review
HTTP logs/testcase/stack (17.57 KB, application/zip)
2008-03-25 23:41 PDT, :Gavin Sharp [email: gavin@gavinsharp.com]
no flags Details
regression fix? (1.68 KB, patch)
2008-03-26 19:53 PDT, :Gavin Sharp [email: gavin@gavinsharp.com]
no flags Details | Diff | Splinter Review
regression fix (4.79 KB, patch)
2008-03-27 17:27 PDT, :Gavin Sharp [email: gavin@gavinsharp.com]
cbiesinger: review+
cbiesinger: superreview+
Details | Diff | Splinter Review
combined patch (7.85 KB, patch)
2008-03-28 15:26 PDT, :Gavin Sharp [email: gavin@gavinsharp.com]
mbeltzner: approval1.9+
Details | Diff | Splinter Review
combined patch + additional fix (8.19 KB, patch)
2008-04-08 14:47 PDT, :Gavin Sharp [email: gavin@gavinsharp.com]
cbiesinger: review+
cbiesinger: superreview+
Details | Diff | Splinter Review
rollup patch (9.34 KB, patch)
2008-04-16 21:07 PDT, :Gavin Sharp [email: gavin@gavinsharp.com]
no flags Details | Diff | Splinter Review
Activity monitor samples during a typical "proxy hang". (6.72 KB, text/plain)
2008-04-28 09:43 PDT, Peter da Silva
no flags Details
rollup patch + hack to set referer header (8.96 KB, patch)
2008-07-09 07:48 PDT, Jeff Muizelaar [:jrmuizel]
no flags Details | Diff | Splinter Review
rollup patch + copy all headers (6.49 KB, patch)
2008-07-09 14:06 PDT, Jeff Muizelaar [:jrmuizel]
cbiesinger: review+
Details | Diff | Splinter Review
breaks lines netwerk/protocol/http/src/nsHttpChannel.h (3.23 KB, patch)
2008-08-05 10:51 PDT, Jeff Muizelaar [:jrmuizel]
no flags Details | Diff | Splinter Review
rollup patch + copy all headers (9.25 KB, patch)
2008-08-05 10:56 PDT, Jeff Muizelaar [:jrmuizel]
cbiesinger: superreview+
Details | Diff | Splinter Review
v2 (9.14 KB, patch)
2008-09-30 15:06 PDT, Jeff Muizelaar [:jrmuizel]
no flags Details | Diff | Splinter Review
Necko unit test (5.14 KB, patch)
2008-10-29 14:38 PDT, Jeff Muizelaar [:jrmuizel]
no flags Details | Diff | Splinter Review
[1/2] break lines in netwerk/protocol/http/src/nsHttpChannel.h (3.04 KB, patch)
2008-11-03 14:51 PST, Jeff Muizelaar [:jrmuizel]
no flags Details | Diff | Splinter Review
[2/2] v3: Defer proxy resolution for HTTP and HTTPS PAC to avoid blocking main thread during DNS resolution (14.15 KB, patch)
2008-11-03 14:57 PST, Jeff Muizelaar [:jrmuizel]
no flags Details | Diff | Splinter Review
[2/2] v4: Defer proxy resolution for HTTP and HTTPS PAC to avoid blocking main thread during DNS resolution (14.17 KB, patch)
2008-11-04 16:50 PST, Jeff Muizelaar [:jrmuizel]
mbeltzner: approval1.9.1b2-
Details | Diff | Splinter Review
"[PAC] Defer proxy resolution for HTTP and HTTPS PAC to avoid blocking main thread during DNS resolution" [a-1.9.1b2=mbeltzner] (23.52 KB, patch)
2012-02-03 07:34 PST, Josh Matthews [:jdm]
no flags Details | Diff | Splinter Review
Defer proxy resolution for HTTP and HTTPS PAC to avoid blocking main thread during DNS resolution. (22.34 KB, patch)
2012-02-03 10:16 PST, Josh Matthews [:jdm]
no flags Details | Diff | Splinter Review

Description Stewart Jeacocke 2004-02-27 10:47:15 PST
User-Agent:       
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7b) Gecko/20040225

This bug has been split from #224447 as the symptoms are different. I will copy
relevant comments from #224447 into this bug report.

Reproducible: Always
Steps to Reproduce:
1. Go to Preferences, Advanced, Proxies
2. Select "Automatic Proxy Configuration URL"
3. Enter a URL to a valid PAC file
4. Click OK
5. Browse a website (particularly one with many offsite adds/images)


Actual Results:  
DNS resolution causes the UI to hang

Expected Results:  
DNS resolution should not cause the UI to hang (it doesn't when using either
"Direct connection to the Internet" or "Manual Proxy Configuration")
Comment 1 Stewart Jeacocke 2004-02-27 10:53:54 PST
Copied from bug #224447

 ------- Additional Comment #6 From mike  2004-01-20 17:17 PST  [reply] -------

Also experiencing this.  Firebird 0.8+ MacOS X
Resolving Host takes about 5 seconds for any page with a new DNS.  For example,
if i go to www.google.com, it will hang for a few secs at resolving host before
loading, but then anything i do within google will be snappy.  But, say i click
a link from google to another site, it will hang for another few seconds until i
can get into that new site's DNS.


------- Additional Comment #7 From emmet@cogs.sussex.ac.uk 2004-02-17 05:38 PST
[reply] -------

This ought to be renamed to "Complete hang when DNS resolving/lookup"

I see this too, but not always (local site dns cacheing?) -- and experienced
this also when using Mozilla. 

Note: IE does not hang -- that is, its menus etc still function, it can redraw
etc while the lookup occurs.


(Firefox 0.8, Win2K, via a proxy squid/2.5.STABLE3)

------- Additional Comment #9 From Stewart Jeacocke 2004-02-25 06:30 PST [reply]
-------

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7b) Gecko/20040224

As of 23/02/04 I'm still seeing this bug on the SeaMonkey CVS Tree. The UI 
(every windows and tab) is blocked while name resolution is performed. This is a 
real PITA particulary for sites which have many adds etc. For example it can 
take over 30s for http://www.ebay.co.uk to load during which time the UI is 
completely hung.

This bug seems to be due to two things
i) The UI is blocked during host resolution
AND
ii) Host resolution in mozilla is sometimes very slow, even when the dns names 
have already been cached at a local DNS server. For example I used the "host" 
command to resolve all the names required for www.ebay.co.uk and none of them 
took longer than 0.05s to resolve


------- Additional Comment #10 From Stewart Jeacocke 2004-02-25 06:42 PST
[reply] -------

This bug is not specific to Firefox. Would it be a good idea to change "Product" 
to something other than FireFox?

I've stuck some cout statements at the beginning and end of nsDNSService::
Resolve() [nsDNSService2.cpp] and can confirm that we are inside this method 
when the UI hangs.

I'm willing to help track down the cause of this bug (as its making me have to 
use Opera ;-) ) but I have no experience with the internals of mozilla so would 
need guidance as to where to start looking.


------- Additional Comment #11 From Stewart Jeacocke 2004-02-25 12:56 PST
[reply] -------

It seems that this bug is triggered by something in my preferences. I moved my
.mozilla directory out of the way and let mozilla create a new one. This caused
the bug to disappear (at least on current CVS and mozilla 1.6).

When using my old .mozilla dir (the config the elicits this bug) the method
nsDNSService::Resolve() is being called lots, however when I let mozilla create
.mozilla from scratch nsDNSService::Resolve() is never called and is replaced by
calls to nsDNSService::AsyncResolve()


------- Additional Comment #12 From Stewart Jeacocke 2004-02-25 14:30 PST
[reply] -------

The offending preference is enabling Proxy Auto Config (PAC).

When I enable PAC (set to use a valid remote PAC file) the
nsDNSService::Resolve() method is used. When I either manually set the proxy
config or select "direct connection" the nsDNSService::AsyncResolve() method is
called.

SUMMARY
Using Proxy Auto Config stops DNS resolution from happening asynchronously. This
causes the UI to hang while name resolution occurs. 


------- Additional Comment #13 From Jason Barnabe 2004-02-25 14:47 PST [reply]
-------

Moving to Browser, good work Stewart.


------- Additional Comment #14 From Jason Barnabe 2004-02-25 14:49 PST [reply]
-------

*** Bug 213751 has been marked as a duplicate of this bug. ***


------- Additional Comment #15 From Stewart Jeacocke 2004-02-27 03:08 PST
[reply] -------

Not sure if this helps but this is a example bracktrace that I get if I set a 
breakpoint on the nsDNSService::Resolve() method (with proxy auto config 
enabled).

#0  nsDNSService::Resolve(nsACString const&, int, nsIDNSRecord**) 
(this=0x8110a50, hostname=@0x889d658, bypassCache=0, resul$
#1  0x4012a025 in XPTC_InvokeByIndex () from ./libxpcom.so
#2  0x4089e3b7 in XPCWrappedNative::CallMethod(XPCCallContext&, 
XPCWrappedNative::CallMode) (ccx=@0xbfffe650, mode=CALL_METH$
#3  0x408a8913 in XPC_WN_CallMethod(JSContext*, JSObject*, unsigned, long*, 
long*) (cx=0x818cce0, obj=0x87fd680, argc=2, arg$
#4  0x401b2770 in js_Invoke (cx=0x818cce0, argc=2, flags=0) at jsinterp.c:941
#5  0x401c0660 in js_Interpret (cx=0x818cce0, result=0xbfffedcc) at 
jsinterp.c:2962
#6  0x401b27ea in js_Invoke (cx=0x818cce0, argc=3, flags=2) at jsinterp.c:958
#7  0x40896e3b in nsXPCWrappedJSClass::CallMethod(nsXPCWrappedJS*, unsigned 
short, nsXPTMethodInfo const*, nsXPTCMiniVariant$
#8  0x4088fd2f in nsXPCWrappedJS::CallMethod(unsigned short, nsXPTMethodInfo 
const*, nsXPTCMiniVariant*) (this=0x88ef1a0, me$
#9  0x4012a396 in PrepareAndDispatch (methodIndex=4, self=0x88ef1a0, 
args=0xbffff214) at xptcstubs_gcc_x86_unix.cpp:100
#10 0x40ad6c23 in nsStreamListenerTee::OnStopRequest(nsIRequest*, 
nsISupports*, unsigned) (this=0x81e8ba0, request=0x8854778$
#11 0x40b780a6 in nsHttpChannel::OnStopRequest(nsIRequest*, nsISupports*, 
unsigned) (this=0x8854778, request=0x88c1898, ctxt$
#12 0x40aae8ca in nsInputStreamPump::OnStateStop() (this=0x88c1898) at 
nsInputStreamPump.cpp:498
#13 0x40aae2e8 in nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*) 
(this=0x88c1898, stream=0x889d4cc) at nsInputS$
#14 0x400d87c7 in nsInputStreamReadyEvent::EventHandler(PLEvent*) 
(plevent=0x8891454) at nsStreamUtils.cpp:118
#15 0x400fc41b in PL_HandleEvent (self=0x8891454) at plevent.c:671
#16 0x400fc2d0 in PL_ProcessPendingEvents (self=0x80bd5a8) at plevent.c:606
#17 0x400ff4ec in nsEventQueueImpl::ProcessPendingEvents() (this=0x80bd570) at 
nsEventQueue.cpp:391
#18 0x416ec720 in event_processor_callback (data=0x80bd570, source=9, 
condition=GDK_INPUT_READ) at nsAppShell.cpp:186
#19 0x416ec0cd in our_gdk_io_invoke (source=0x81ec768, condition=G_IO_IN, 
data=0x81ee1f0) at nsAppShell.cpp:71
#20 0x4042fa56 in g_io_add_watch () from /usr/lib/libglib-1.2.so.0
#21 0x4043103d in g_get_current_time () from /usr/lib/libglib-1.2.so.0
#22 0x404314f4 in g_get_current_time () from /usr/lib/libglib-1.2.so.0
#23 0x40431724 in g_main_run () from /usr/lib/libglib-1.2.so.0
#24 0x40358c3f in gtk_main () from /usr/lib/libgtk-1.2.so.0
#25 0x416ecb4e in nsAppShell::Run() (this=0x811dca0) at nsAppShell.cpp:317
#26 0x4169c4cc in nsAppShellService::Run() (this=0x811d408) at 
nsAppShellService.cpp:483
#27 0x080570e9 in main1 (argc=1, argv=0xbffff744, nativeApp=0x809e540) at 
nsAppRunner.cpp:1291
#28 0x08057d16 in main (argc=1, argv=0xbffff744) at nsAppRunner.cpp:1678

Comment 2 Jason Barnabe (np) 2004-02-27 11:29:58 PST
Confirming on Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.6)
Gecko/20040206 Firefox/0.8
Comment 3 Chase Tingley 2004-02-27 14:06:34 PST
This is a dup of a bug that was WONTFIX'd about a year and a half ago.  The
problem is that DNS resolution run in a PAC file is done in the UI thread,
whereas all normal DNS resolution is done in the network thread itself.  I'll
look up the original bug; I seem to remember it being an issue of ugly
architectural issues.
Comment 4 Chase Tingley 2004-02-27 14:09:35 PST
It was bug 97605 (the actual resolution was invalid).  The "workaround" is not
to use PAC scripts that use dnsResolve().
Comment 5 Stewart Jeacocke 2004-02-28 12:00:29 PST
The PAC script in question doesn't use dnsResolve(). But it does use
isResolvable() which I guess may be the culprit. Here is the PAC function

function FindProxyForURL(url, host)
{
  //So the error message "no such host" will appear through the
  //normal Netscape box - less support queries :)
  if (!isResolvable(host))
    return "DIRECT";
 
  if (isPlainHostName(host) ||
      dnsDomainIs(host, ".lan") ||
      dnsDomainIs(host, ".jeacocke.org.uk"))
    return "DIRECT";
  else
    return "PROXY hadron:3128; DIRECT";
}
Comment 6 benc 2004-04-16 05:21:00 PDT
fixed blocks, added depends.

isResolvable() calls dnsResolve()

http://lxr.mozilla.org/mozilla/source/netwerk/base/src/nsProxyAutoConfig.js#267

267 "function isResolvable(host) {\n" +
268 "    var ip = dnsResolve(host);\n" +

Chase, is this a won't fix?
Comment 7 Chase Tingley 2004-04-17 12:26:10 PDT
I'll leave that for someone from Networking to decide, but I think it's a likely
outcome.  I'd really like to see it fixed, but last time it was discussed, it
seemed like an arch problem in the worst way.
Comment 8 benc 2004-05-10 09:31:08 PDT
The reason I'm asking is that I'm about to roll up some network admin docs for
PAC, so if you mark this wontfix, I'll use that as the final state of the bug.
Comment 9 Jason Barnabe (np) 2004-11-22 09:53:07 PST
Are there any workarounds for this? (For example, changing a setting so that the
resolution will give up sooner)
Comment 10 LaPlaine 2005-01-07 07:38:04 PST
This bug is killing me!!! Simply type an URL that won't resolve and all tabs are
locked permentatly. I need to kill process and start all over again. Easily
repeatable on XP running Firefox 1.0

There are some screwing things going on with DNS here, can we please get this
fixed before I'm forced to abandon this browser
Comment 11 benc 2005-01-14 08:47:23 PST
Jason: I was reading DNS&BIND the other day for work, and I think some very
modern BIND resolvers allow system-level timeout settings.

The other question is: why are you using this in the PAC file. When I supported
PAC for Netscape, I found only one customer that did this for a good reason, and
that was predicated on a completely bad DNS subdomain strategy that they decided
to re-do, based on this bug.
Comment 12 Mark 2005-02-15 07:54:10 PST
Confirm this problem occurs in FF 1.0 release under both XP and Gentoo ith FF 1.0.

The same effect is not noticed in IE, although obviously with it not using tabs
that could just be me not noticing it due to the expectation of a **** UI.

In my case I am using the automagically detect my proxy setting, rather than the
PAC file. The result is the same in that the UI appears to die toally :(


Comment 13 Peter Janes 2005-05-30 18:55:21 PDT
benc in comment 11:

> The other question is: why are you using this in the PAC file.

VPN.  Specifically, using isInNet(host,subnet,mask) to proxy only certain
address blocks, the inverse of "No Proxy for" that can't easily be accomplished
any other way.

To see the full effect of this bug, by the way, try browsing around
http://www.myspace.com/ for a while with a few tabs open.  Ouch.

Firefox 1.0.4 on Linux, ditto for recent trunk builds.
Comment 14 Darin Fisher 2006-02-21 17:39:39 PST
Can those of you who experienced this bug or are still experiencing this bug, please comment about the behavior with FF 1.5?  I did some work on FF 1.5 that should have improved things for the case where the DNS query is made for the hostname passed to FindProxyForURL.  Thanks!
Comment 15 Jason Barnabe (np) 2006-02-21 17:56:17 PST
Haven't noticed any change. Still hangs for at least a minute.
Comment 16 Peter Janes 2006-02-21 18:57:40 PST
Same here as comment 15.
Comment 17 Darin Fisher 2006-03-09 22:04:00 PST
Jason and Peter, can either of you share your PAC file with me?  If you could post it to this bug report that would be ideal.  If there is anything sensitive in there, feel free to replace it with random characters or whatever... I just want to get a sense for the kind of function calls made by your PAC scripts.  Thanks!
Comment 18 Peter Janes 2006-03-09 22:20:33 PST
Pretty straightforward as far as I can tell:

function FindProxyForURL(url, host)
{
        if(
                isInNet(host,"192.168.15.0","255.255.255.0") ||
                isInNet(host,"10.120.0.0","255.255.0.0") ||
                isInNet(host,"10.180.0.0","255.255.0.0")
        )
        {
                return "PROXY proxy:8080";
        }
        else
        {
                return "DIRECT";
        }
}

I have another one that basically reverses the DIRECT and PROXY return values.
Comment 19 Jason Merrill 2006-03-10 11:05:59 PST
I no longer have the PAC file I was having this trouble with; I worked around it by just proxying based on the domain name rather than trying to be clever about only proxying unresolvable hosts.
Comment 20 Miko Brown 2006-03-18 20:59:46 PST
Here is my pac file.  Right now I have some debug code in there to show that there is a several second delay when resolving a new DNS name but the delay is there regardless if I have dnsResolve there or commented out.  So it almost seams like it is doing a DNS lookup in the UI code.  Also adding my debug code does not slow it down any.  So if I have to guess I would say that there is a DNS lookup sometime after the function FindProxyForURL is called.

PS I am using FF 1.5.0.1 on Win XP.

function FindProxyForURL(url,host)
{


//#######
//#Debug#
//#######

var time = new Date();
sec1=time.getSeconds();
milli1=time.getMilliseconds();

remoteIP=dnsResolve(host);

var time2 = new Date();
sec2=time2.getSeconds();
milli2=time2.getMilliseconds();

alert(host+" Resolved to "+remoteIP+" in "+(sec2-sec1)+":"+(milli2-milli1));


//###############
//#Local Network#
//###############

if (isInNet(host, "10.0.0.0", "255.0.0.0")){
            return "DIRECT";
}

//#########
//#Library#
//#########

if (	host == "auth.somesite.com" ||
	host == "search.someothersite.com" ||
	host=="yetanothersite.com" ){
	return "DIRECT";
}


//#########
//#Default#
//#########

return "PROXY proxy.ourdomain.com:8000";

}

Comment 21 Darin Fisher 2006-03-20 11:13:33 PST
Have you changed the "network.dnsCacheExpiration" preference?
Comment 22 Darin Fisher 2006-03-20 11:16:33 PST
Ah, nevermind.  I know what the problem here is.  We need to teach the IOService + HTTP protocol handler how to use nsIProtocolProxyService::AsyncResolve.  If we do that, then the code in nsPACMan.cpp that pre-resolves the URL's hostname will help eliminate this bug for cases where the given hostname is resolved by the PAC script.
Comment 23 Darin Fisher 2006-03-20 11:18:11 PST
We already have code in the HTTP channel that uses AsyncResolve during startup before the PAC script is loaded, and we might be able to extend that to work here.  Not sure yet.
Comment 24 Daniel Mota Leite 2006-04-09 09:43:12 PDT
any change of being fixed still in versions 1.x?

we want to deploy firefox for all company, but we use a .pac to bypass the proxy when it resolves to a local IP, that is hard to workaround by domain names because a we have a division that develop sites, so they always have new domains... keeping add this new domains for firefox to work well will be a hell...

So this bug seens like a blocking bug for us...

my .pac is basically close to comment #18
Comment 25 Adam Pribyl 2006-11-01 07:39:29 PST
Is not fixed even in 2.0... :(
Comment 26 Boris Zbarsky [:bz] 2006-11-01 07:43:31 PST
It's clearly not fixed -- the bug is not "RESOLVED FIXED".

Someone might need to step up and do what Darin suggests...
Comment 27 Adam Pribyl 2006-12-18 08:32:18 PST
The UI is also freezing when FF is first connected to network that uses PAC proxy, but then you change the network to one without PAC proxy. Then you have to wait very long time before FF finds there is no proxy.
The cause of this is probably same.
Comment 28 Christopher C. Ritchey 2007-01-10 09:20:05 PST
(In reply to comment #26)
> It's clearly not fixed -- the bug is not "RESOLVED FIXED".
> 
> Someone might need to step up and do what Darin suggests...
> 

I can confirm that this hasn't been fixed in 2.0.0.1 and has been bugging me for the last 2 years...

I'd love to work on this bug but I've never done any development for mozilla or the open source, but yes I know how to program. So I'm willing to make a stab but can't make any promises on success or follow through.
Comment 29 Aaron Sloman 2007-01-21 06:27:23 PST
I think I have been a victim of this bug for several months, using Firefox
on linux (currently Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.8.1) Gecko/20061010 Firefox/2.0 running on Fedora Core 6).

When I work on this machine at home, using cable service provider, everything works fine most of the time.
But occasionally I use the the 'automatic proxy configuration option' to go via the proxy server at my university in order to gain access to research stuff on the web. For some web sites everything works, but a little more slowly.

Sometimes however, the delay is excessive, and then everything concerned with firefox freezes. I cannot change tabs, abort the lookup, or open a new window. Then either I have to wait until the web page is retrieved, or the lookup fails. That can take so long that I sometimes end up having to kill firefox, which can be a real pain, as it sometimes has has several windows and tabs open recording things I was doing.

If, as someone else said, Opera does not have this problem, then presumably it should be fixable in Firefox. I presume it means that all lookups have to be
done in such a way that other interactions are never blocked.

Sorry I can't help with debugging etc. I try to help with promoting FF,
but will have to be cautious about doing that while this bug persists
as it is disastrous.

Aaron
http://www.cs.bham.ac.uk/~axs
Comment 30 Steve Deal 2007-01-23 05:19:37 PST
<p>It sound like Darin <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=235853#c22">comment #22</a> has a through understanding of the problem, the solution and the code. While some of us might take a try at fixing it, I have the impression that Darin (and probably others) know how to implement the solution quickly and properly. </p>
<p>I voted for it as a bug, you should <a href="https://bugzilla.mozilla.org/votes.cgi?action=show_user&bug_id=235853#vote_235853">vote</a>  if you want it fixed.</p>
<p>Bugs like this definitely keep Firefox from being accepted much less being adopted in the corporate world...</p>
Comment 31 Boris Zbarsky [:bz] 2007-01-23 08:28:28 PST
Steve, Darin is not really working on this stuff now.  And please, no HTML in comments.  It makes them _very_ hard to read with all those '<' and '>' all over. ;)
Comment 32 Steve Deal 2007-01-23 09:33:34 PST
My apologies to everyone about the HTML in the posting.  I saw href links in previous postings and attempted to provide the same for convenience.
I couldn't find the link for the "preview" page :-)
Comment 33 Benjamin Smedberg [:bsmedberg] 2007-02-01 07:14:43 PST
Not a blocker, but we'll take a patch for 1.9 if it appears.
Comment 34 Sergio Correia 2007-02-02 12:22:17 PST
I hate this bug with all the strength of my soul. Heck, I'm even considering quitting my job, buying a C++ book, learning it, finding where the bug is, and patching it. That, or switching to opera.

Please, VOTE for the bug to be fixed.
Comment 35 Jason Barnabe (np) 2007-02-02 12:47:50 PST
As a workaround, what I did was download the PAC script I use, commented out lines containing isInNet and sites I never visit, saved it locally, and configured Firefox to read from my local file.
Comment 36 Peter Ford 2007-02-22 10:20:55 PST
I use FF 2.0.0.1, and my UI is hanging a lot, with a similar (but not the same) cause and symptoms.  Please let me know whether you think I should open a new bug.

The cause is that the company I work for now requires all Internet access to be through a proxy, and so I have entered the proxy's details in the "Manual proxy configuration" section of the Connection Settings (as opposed to "Automatic Proxy Configuration URL" which this bug has been about so far).

The symptoms are that most Web page loads cause the UI to hang, even if I have just loaded a page from the same website (as opposed to only pages "with a new DNS", which is what Comment #1 says).

It may be that this is exactly the same underlying issue as reported above but that the proxy I have to use is poorer quality than most.  But I don't think the UI should hang even if the proxy is causing trouble.

Typically I will click on a link or type in an address, and then have to wait from 5 seconds to 5 minutes before I get my UI back.  This hang time may include very brief lapses - just enough for most of the page to load, for example.

Sometimes a single website will start to work properly for a while, with pages loading quickly and no hanging at all, but it could at any moment regress and start hanging again.


The only entries in my "No Proxy for" section are 'localhost', '127.0.0.1', and one other specific local address, and yet most of my company's websites (both local and public), such as www.atosorigin.com do not have this hanging problem.  So presumably the proxy is configured to treat these sites differently (or maybe they're just faster?), and this is being reflected in FF's behaviour.


For the record, I found bug #306922, which seems similar, and has a comment that it may be a duplicate of this one.

I also looked at bug #224447 (from which this one split) and its chain of duplicates bug #188332 and bug #240759, but I don't know how much use those are.
Comment 37 Daniel Mota Leite 2007-02-22 14:11:24 PST
Peter: i think your problems maybe related to some extension, as it seens a DNS problem...

you probably dont have access to a internet capable DNS server, only your internal DNS and maybe some extensions is trying to resolve the hostnames all the time...
try to use firefox in safe mode
also, try to ping a server and see how much time it takes to resolve (or if resolves)

other ideia might be the 127.0.0.1... it might force firefox to resolve the hostname to check if it is 127.0.0.1... try to put only hostnames, no IPs
Comment 38 Mihails Bogdanovs 2007-02-23 03:58:26 PST
It is a standard practice in our company to use Automatic Proxy Configuration especially because of the VPN and many company locations.
And it's a pity for such a good browser to hang all browser instances and tabs if by mistake a wrong URL was typed in the address bar.
I have no such problem with Opera for example.
Comment 39 Peter Ford 2007-02-23 07:28:17 PST
Thanks Daniel for your suggestions.  The behaviour has now changed; hopefully this will let us work out exactly what is causing the problem.  I will investigate futher and report back here.
Comment 40 Peter Ford 2007-03-06 03:20:47 PST
Actually it seems the change of behaviour I reported in Comment #39 was a co-incidence.  I have removed all entries from the "No Proxy for" section of my connection settings, and I now normally use Safe Mode (even though the only add-ons I have are the DOM Inspector, Talkback, the British English Dictionary and the default Firefox Theme).  Also I am now at 2.0.0.2.

None of these factors seems to make any difference to the problem, which is actually intermittent.  Occasionally there are periods where everything works fine (and hence my Comment #39), but 95% of the time I get the hanging.

So I presume the problem is related to the proxy.  But I don't think the UI should hang - even if the proxy is causing trouble.


If anyone isn't clear, I have screenshots to show what actually happens.  I can also supply some full ping results (as suggested in Comment #38), but in summary I think they show that DNS is working ("Pinging recluse.mozilla.org [63.245.208.164]") and that my company is blocking ping ("Destination net unreachable." or "Request timed out.")


Do you think this is the same underlying issue as reported above?  (and described in Comment #3 as "ugly architectural issues")  Or shall I open a new bug?  I would be glad to investigate this further in conjunction with someone who knows what they're doing; I could run debuggers, system performance monitors, or anything that would help.  
Comment 41 Anthony Guselnikov 2007-06-06 08:58:35 PDT
Created attachment 267425 [details]
Improper connection settings UI 

There is one other thing related to this bug. Sometimes when this proxy-hang happens firefox crashes, and the Network Connection Settings UI becomes unusable - see attached screenshot. Each time this happens I simply re-install firefox and it comes back to normal. It'd be nice if when network connection info is compromised UI didn't hose itself.
Comment 42 Sascha Schmidt 2007-06-19 05:24:32 PDT
I come from bug: https://bugzilla.mozilla.org/show_bug.cgi?id=309582 where the same problem is discussed. We adjure that this bug will be solved in the NEAR future. This problem is annoying me for a VERY long time now!

Bug is still active on:
Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.8.1.4) Gecko/20070515 Firefox/2.0.0.4
Comment 43 Sascha Schmidt 2007-06-19 07:42:43 PDT
I really found a workaround for this problem:

Use your wpad.pac file like this:

function FindProxyForURL(url, host)
{
   if (isResolvable(host)
   {
       <... your stuff ...>
   } else
      return "DIRECT";
}

This works for me... 

Greets
Sascha
Comment 44 Michael Mauch 2007-06-20 10:02:22 PDT
Thank you, Sascha! Your workaround works for me. 

function FindProxyForURL(url, host)
{
        if(!isResolvable(host))
                return "DIRECT";

        <... your stuff ...>
}

The only remaining problem is the fact that "<... your stuff ...>" is normally pulled from a web server, and it might change. Apparently there is no include statement for these proxy.pac files, so I would need to find someone who could change the official company proxy.pac. And then tell them to look at this bug and please change that pac file accordingly.

So, could something like this be put into the Firefox source? Try to resolve the hostname just before calling that FindProxyForURL() function?

Or wouldn't that help?

Regards...
  Michael
Comment 45 Jason Merrill 2007-06-22 20:42:55 PDT
(In reply to comment #44)

That default wouldn't work because, for instance, I would like to use a proxy only for unresolvable hosts (because they are internal hosts that I need to access through an ssh-port-forwarded proxy).
Comment 46 Artem Marchenko 2007-07-20 06:30:57 PDT
Particularly difficult case for this bug - http://symbiansmartphone.blogspot.com/ 
It usually takes some 30 secs to load on my mega-powerful double Xeon 2GHz with 2 gigs of ram. And most of these 30 secs the Firefox is not responsive.

Originally I blamed http://symbiansmartphone.blogspot.com/ but now I see that it is slow PAC file processing that is guilty.
Comment 47 Samuel Sidler (old account; do not CC) 2007-08-11 15:31:54 PDT
*** Bug 391682 has been marked as a duplicate of this bug. ***
Comment 48 Jeremy Baron 2007-10-02 16:39:25 PDT
downstream dupe with some more comments:
https://bugs.launchpad.net/ubuntu/+source/firefox/+bug/113201
Comment 49 Oles Hnatkevych 2007-10-23 23:48:02 PDT
I wonder if it will ever be fixed. Seems like devs completely ignore this issue.
Comment 50 Jilles van Gurp 2007-11-23 08:37:27 PST
Just for the record, this bug is affecting Firefox 3.0 Beta 1 as well. I tried it today and the UI completely blocks when it is accessing the proxy. Again, this will likely affect any enterprise users behind proxies with proxy scripts. As far as I know that would be just about any fortune 500 company.

Neither internet explorer 6 nor safari 3.0.4beta have this issue on my laptop. Nor does the problem surface when browsing outside the corporate intranet. 

Worst case for this bug is pages with content from various sites that each need to be resolved by the script. I've seen the browser block UI for 10-20 seconds on such sites which is of course totally unacceptable.  

I'm afraid this bug has joined the ranks of ignored bugs. If a moz developer is reading; please do something with this bug because this makes Firefox suck big time in Nokia.
Comment 51 Dr. David Alan Gilbert 2007-11-24 07:28:55 PST
I agree this really needs some work; the problem can be worked around by reworking your proxy.pac to do everything using regexp on the names rather than resolves; but this is a big job and can end up with a very complex pac depending on your network.

It also means that when dropping Firefox into an organisation which is using something else it makes Firefox look really bad, and since it isn't obvious it is a .pac issue it just means Firefox looks slow and clunky.

Dave
Comment 52 Daniel Mota Leite 2007-11-24 16:30:11 PST
when i changed work, i arrived to a company that used a .pac that blocked firefox and because of this, almost no one used firefox...

i fixed the .pac and right now, most web developers use firefox (a great thing for compatibility of our sites) and several people in all departments started using firefox.

this is great, but only the admins could change the .pac and none of then knew about this problem, or even cared about it, so all users couldnt do anything to fix this, other than download the pac and fix it... of course, almost no one know how to do it.

this bug is a big problem because only companies and ISPs use .pac files, and user that use this pac arent tecnical enough to find and workaround the problem.
in the end, this people will not use firefox because it is slow.

waiting for ISPs and companies to fix their .pac will take ages, assuming that it will be fixed at all!!
+ wanted1.9 would be nice, but this needs to be fixed, the sooner, the better
Comment 53 Aaron Sloman 2007-11-28 02:15:38 PST
(In reply to comment #52)

> this bug is a big problem because only companies and ISPs use .pac files

Not so: I work in a university (www.cs.bham.ac.uk) and we have the same problem.

This bug seems to me to have much higher priority than some other things that have been changing in firefox.

Unfortunately I don't have the technical knowledge to work on it.

Aaron
Comment 54 Wayne Mery (:wsmwk, NI for questions) 2007-11-28 06:23:54 PST
is bug 306922 a dupe as suggested in it's comment 11?
Comment 55 RalfG 2008-01-09 11:08:10 PST
I can confirm this bug, using Firefox 2.0.0.11 for windows or Iceweasel for Debian in combination with a Debian Edu/Skolelinux setup (WPAD on Tjener, auto detection of proxy set on). Slightly disturbing for power users that are used to immediate reaction of firefox. I agree that this bug is of high priority.
Comment 56 timeless 2008-01-22 05:22:43 PST
the only fix i can see is moving pac resolving off the main thread. JavaScript is not in any particularly good position to store state so that a caller can resume execution when it has the answer it needs. OTOH, if the JavaScript runs on a different thread, this problem magically goes away.

ralfg: if it's of high priority to you, then either write a patch, or find someone who's willing to accept your payment to write a patch.

You're supposed to read "Bugzilla Etiquette" <https://bugzilla.mozilla.org/page.cgi?id=etiquette.html>, linked from <https://bugzilla.mozilla.org> before commenting in bugs.
Comment 57 Jason Faulkner 2008-01-31 07:37:18 PST
Confirming that this bug still exists on Firefox 3 Beta 2 for Windows Vista.
Comment 58 abe 2008-02-04 16:54:41 PST
it's a fairly annoying bug that prevent many corporate users from using mozilla unfortunately.
As Dr. David and Daniel mentioned, there is a workaround by fixing the pac file.  wonder if anyone is kind enough to write an example for this workaround?... before the bug is fixed hopefully...  thanks
Comment 59 Jason Faulkner 2008-02-04 18:07:01 PST
The workaround does /not/ work. It only prevents *nonexistent* domains from slowing down Firefox. Any site which has slow DNS (or if DNS servers for your network are slow) will still lock the UI until it resolves. 
Comment 60 Oles Hnatkevych 2008-02-04 23:01:42 PST
function FindProxyForURL(url, host)
{
        if (dnsDomainIs(host, ".mydomain.com"))
            return "DIRECT";
        else
        if (isPlainHostName(host))
            return "DIRECT";
        else
        if (host == "localhost")
            return "DIRECT";
        else
        if (shExpMatch(host, "127.*"))
            return "DIRECT";
        else
        if (shExpMatch(host, "10.0.0.*") )
            return "DIRECT";
        else
            return "PROXY 10.0.0.1:3128";
}

This way no DNS queries are done during proxy resolution.
Comment 61 Daniel Mota Leite 2008-02-06 12:16:30 PST
correct, the posted workaround works fine, Jason may have tried a bad workaround, using this was the dns code that locks the firefox doesnt run. any site with slow or no dns will be slow, but JUST for that tab, not all firefox like this bug.

i use a little more elaborate code, but might be useful for those with less proxy.pac knowledge:

if ( isPlainHostName(host)                  ||
      shExpMatch(host, "*localhost*")        ||
      shExpMatch(host, "*producao*")         ||
      shExpMatch(host, "*internal-tests*")   ||
      shExpMatch(host, "*intranet-pe*")      ||
      dnsDomainIs(host, "domain1.pt")        ||
      dnsDomainIs(host, "domain2.pt")
   ) { return "DIRECT"; }
else if ( shExpMatch(url, "*10.10.*")       ||
          shExpMatch(url , "*:7778*")       ||
          shExpMatch(url, "*127.0.0*")      ||
          shExpMatch(url, "*192.168.*")
      ){ 
         if ( isInNet(host, "10.10.0.0", "255.255.0.0")     ||
              isInNet(host, "127.0.0.0",  "255.0.0.0")      ||
              isInNet(host, "192.168.0.0", "255.255.0.0")
            ) { return "DIRECT"; }
         else { return "PROXY proxy:3128"; }
      }
else { return "PROXY 10.10.0.241:8080"; }

note that my code DOES check for the DNS, but only if the URL have some kind of IPs... i do this because i have a internal server with a IP in the URL (not in the host part), so i assume that might also happens in the net... during this years i only found one public server with a url with a internal IP.
neverless, i have a internal DNS that resolves those private IPs, so i get no slowdown... but you can remove that isinet code and it works fine also.

higuita
Comment 62 georgi - hopefully not receiving bugspam 2008-03-07 03:06:17 PST
this bug turned in search for "wpad"

seamonkey trunk tries to resolve |wpad.localdomain| on visiting any url - checked with sniffer on localhost.

is this normal?

if it resolves because of malicious name server will it override proxy prefs - in this case bug should be filed.
Comment 63 Fedora fonts special interest group 2008-03-07 03:24:25 PST
(In reply to comment #62)
> this bug turned in search for "wpad"

But it's not about the wpad protocol. Open another issue if you care about it
(look up WPAD on wikipedia to see what's it about before)
Comment 64 georgi - hopefully not receiving bugspam 2008-03-07 06:05:39 PST
sorry for the bugspam. the new stuff is Bug 421490
Comment 65 Mike Shaver (:shaver -- probably not reading bugmail closely) 2008-03-24 10:12:32 PDT
Created attachment 311399 [details] [diff] [review]
resolve non-blocking, and if we block defer until later

biesi pointed me in this direction.  I'm spinning a try server build shortly, and writing tests, just throwing it up here for now.
Comment 66 Mike Shaver (:shaver -- probably not reading bugmail closely) 2008-03-24 14:23:45 PDT
Comment on attachment 311399 [details] [diff] [review]
resolve non-blocking, and if we block defer until later

This passes a bunch of manual testing (including network removal, proxy being down, etc.) and the PAC parts of the existing necko tests.  I'd like to get this into b5, and I'll work on unit tests for the specific newChannel cases later this week!
Comment 67 Christian :Biesinger (don't email me, ping me on IRC) 2008-03-24 14:37:48 PDT
Hrm, so... this patch means that for FTP (and Gopher ;) ) we wouldn't use a proxy when PAC is selected. That doesn't seem too good, should we only pass the nonblocking flag when scheme is HTTP/HTTPS?
Comment 68 Mike Shaver (:shaver -- probably not reading bugmail closely) 2008-03-24 14:38:56 PDT
Comment on attachment 311399 [details] [diff] [review]
resolve non-blocking, and if we block defer until later

I don't think this works as we want -- it seems to skip PAC entirely for some protocols.
Comment 69 Mike Shaver (:shaver -- probably not reading bugmail closely) 2008-03-24 15:01:59 PDT
Created attachment 311445 [details] [diff] [review]
With fix to newProxyInfo, and restriction to http/https

This is better, though ftp and gopher can still lock up the UI!

Discovered that I was running the wrong tree when I was working on a unit test for the deferral case -- test isn't working yet, but running the right code sure does work better!

I'll spin up a tryserver with these so that interested folks can report as to their results with slow DNS.  (I'm not quite sure how I'm going to do the slow-DNS testing, other than just going to the office tomorrow!)
Comment 70 Christian :Biesinger (don't email me, ping me on IRC) 2008-03-24 15:15:34 PDT
Comment on attachment 311445 [details] [diff] [review]
With fix to newProxyInfo, and restriction to http/https

this does not actually fix non-HTTP
Comment 71 Mike Shaver (:shaver -- probably not reading bugmail closely) 2008-03-24 19:29:45 PDT
Created attachment 311500 [details] [diff] [review]
Better fix for protocol restriction

This still needs tests, and more test_ing_.  People who can build with this patch and test in their DNS-sensitive proxy configs should consider themselves Cordially Invited to do so and report their findings.
Comment 72 Christian :Biesinger (don't email me, ping me on IRC) 2008-03-25 05:54:36 PDT
Comment on attachment 311500 [details] [diff] [review]
Better fix for protocol restriction

r=biesi with the compile error fixed ;)
Comment 73 Artem Marchenko 2008-03-25 05:59:32 PDT
> People who can build with this patch and test in their DNS-sensitive proxy 
> configs should consider themselves Cordially Invited to do so and report their 
> findings.

I don't really know how to apply patches and compile Firefox and frankly don't have time for it. However, if somebody can tell me where to get a patched binary, I can test it with my office network, where the issue can be reproduced easily all the time.
Comment 74 Mike Shaver (:shaver -- probably not reading bugmail closely) 2008-03-25 06:01:35 PDT
Comment on attachment 311500 [details] [diff] [review]
Better fix for protocol restriction

I'll test this as best I can, and cross-my-heart-hope-to-be-flamed-by-Brad promise to write Tunit fodder once we get clear of the B5 cruch, but I think the bestest of testests is going to be PAC in the wild -- both to determine to what extent it repairs the problem, and to find cases where it might interfere (though I think those unlikely, based on my growing understanding of this code).
Comment 75 Mike Shaver (:shaver -- probably not reading bugmail closely) 2008-03-25 06:18:15 PDT
(In reply to comment #73)
> I don't really know how to apply patches and compile Firefox and frankly don't
> have time for it. However, if somebody can tell me where to get a patched
> binary, I can test it with my office network, where the issue can be reproduced
> easily all the time.

The try server has been activated, and there will be builds available in 30-180 mins (Mac first, Windows last).

https://build.mozilla.org/tryserver-builds/?C=M;O=D

You will want the builds that identify themselves as "nonblocking_pac2", from March 25th.  And I will want to hear about your results!
Comment 76 Mike Shaver (:shaver -- probably not reading bugmail closely) 2008-03-25 06:32:05 PDT
Created attachment 311573 [details] [diff] [review]
posterity: from rediffing after removing the NS_ASSERTION, not just rebuilding and running :(

Addressing biesi's comments, looking for some approval love.
Comment 77 Mike Shaver (:shaver -- probably not reading bugmail closely) 2008-03-25 07:12:14 PDT
https://build.mozilla.org/tryserver-builds/2008-03-25_06:12-shaver@mozilla.com-nonblocking_pac2/ has builds now, please report your findings!
Comment 78 Artem Marchenko 2008-03-25 08:09:31 PDT
Quick testing results:

1. Tried the usual difficult page http://symbiansmartphone.blogspot.com/ in my normal Firefox 2.0.0.12 - not 30 secs of frozing as it used to be several months ago, but still many times browser was frozen during the page loading totally for about 5-10 secs. So bug still could be reproduced

2. Closed normal Firefox, opened the Windows build from the zip file, went to the same page - everything was blazing fast, no freezing at all, not even for 0.5 second

3. Could not believe that it was that fast, closed your build, went back to normal Firefox, opened the same page. It was fast! There were minor freezings (like 0.3-0.5 secs) so your build was faster with absolutely no freezing, but that was way faster than before. Not sure how to explain this (maybe some caching issue?).

I am eager to do more testing, but I am traveling for a couple of days and not sure, when exactly I'll be back in the office. So, feel free to put more "testing tasks" to this thread, but don't expect quick reply (from me at least).

Good luck!
Artem.
Comment 79 Mike Shaver (:shaver -- probably not reading bugmail closely) 2008-03-25 08:27:01 PDT
DNS caching is definitely in play, since the blocking that happens is due to us waiting on a DNS response.  Purging your DNS cache before step 2 would make the results more useful.  Sorry  I didn't say so earlier, if you have a chance to test again with that it would be helpful.  (Or you can go to a "fresh" host first, since the loading speed of the page isn't going to be affected here, just whether we have the browser locked up while waiting for the DNS response in proxy autoconfig.  Comparing vs. Firefox 3 b4 rather than 2.0.0.12 will also probably isolate things better.)
Comment 80 Mike Connor [:mconnor] 2008-03-25 08:30:01 PDT
Comment on attachment 311573 [details] [diff] [review]
posterity: from rediffing after removing the NS_ASSERTION, not just rebuilding and running :(

Would rather risk this in beta than in final, and its a pretty nasty bug for those affected...
Comment 81 Mike Shaver (:shaver -- probably not reading bugmail closely) 2008-03-25 08:58:24 PDT
Checking in base/src/nsIOService.cpp;
/cvsroot/mozilla/netwerk/base/src/nsIOService.cpp,v  <--  nsIOService.cpp
new revision: 1.206; previous revision: 1.205
done
Checking in base/src/nsProtocolProxyService.cpp;
/cvsroot/mozilla/netwerk/base/src/nsProtocolProxyService.cpp,v  <--  nsProtocolProxyService.cpp
new revision: 1.76; previous revision: 1.75
done

May all your PACing be async. Big thanks to biesi for his guidance and review!
Comment 82 :Gavin Sharp [email: gavin@gavinsharp.com] 2008-03-25 11:41:44 PDT
I backed this out due to mochitest failures on 3 platforms. Running mochitest locally, I noticed that it got stuck running
http://lxr.mozilla.org/seamonkey/source/docshell/test/test_bug413310.html

with a "confirm resending POSTDATA" alert. This would be treated like a hang by mochitest and is consistent with the errors on tinderbox (general mochitest FAIL on Linux/Mac, red on Windows).
Comment 83 Artem Marchenko 2008-03-25 11:46:15 PDT
> Purging your DNS cache before step 2 would make the
> results more useful.

Could you tell me how to purge the cache?
Comment 84 Neil Lofland 2008-03-25 11:56:53 PDT
Tested the Trunk Build (3/25/08) with a PAC. The UI froze indefinitely with an invalid URL. I used the invalid URL "http://www.del.icio.us". Valid URLs worked fine. 
shaver@mozilla.com-nonblocking_pac2-firefox-try-win32.zip
WIN XP PRO, SP2
Comment 85 :Gavin Sharp [email: gavin@gavinsharp.com] 2008-03-25 12:22:50 PDT
(In reply to comment #82)
> Running mochitest locally, I noticed that it got stuck running
> http://lxr.mozilla.org/seamonkey/source/docshell/test/test_bug413310.html
> 
> with a "confirm resending POSTDATA" alert.

I've confirmed that backing out the patch fixes that problem.
Comment 86 Boris Zbarsky [:bz] 2008-03-25 12:45:02 PDT
The only reason I can think of for this to affect bug 413310 is if the POST response is not being cached for some reason.  Does someone want to try applying this patch and seeing whether your can submit a POST form (in a subframe, so there's no bfcache), then go back then go forward?  If you get the repost dialog,  repeat all that with HTTP logging enabled?
Comment 87 georgi - hopefully not receiving bugspam 2008-03-25 12:55:05 PDT
hm, how do http proxies deal with the copyrighted stanford pdf trick of changing dns stuff?
Comment 88 Mike Shaver (:shaver -- probably not reading bugmail closely) 2008-03-25 13:55:09 PDT
I don't quite follow what that has to do with this bug?  Do you think that the DNS is changing between resolution points?
Comment 89 :Gavin Sharp [email: gavin@gavinsharp.com] 2008-03-25 19:39:31 PDT
(In reply to comment #86)
> The only reason I can think of for this to affect bug 413310 is if the POST
> response is not being cached for some reason.  Does someone want to try
> applying this patch and seeing whether your can submit a POST form (in a
> subframe, so there's no bfcache), then go back then go forward?

I tested with:
data:text/html,<frameset><frame src="data:text/html,<form method='post' action='http://gavinsharp.com/tmp/echo.php'><input type='hidden' value='foo' name='foo'><input type='submit'>">

With the patch applied, clicked "Submit", clicked back, clicked forward. No POST dialog.
Comment 90 Boris Zbarsky [:bz] 2008-03-25 20:03:52 PDT
But on the mochitest you do see the dialog?
Comment 91 :Gavin Sharp [email: gavin@gavinsharp.com] 2008-03-25 20:42:07 PDT
(In reply to comment #90)
> But on the mochitest you do see the dialog?

Yes.
Comment 92 :Gavin Sharp [email: gavin@gavinsharp.com] 2008-03-25 23:32:44 PDT
I minimized the mochitest to the smallest file that would reproduce the dialog when the patch is applied (to avoid extra loads of JS/CSS), and then generated HTTP logs with/without the patch. All the files are at:

http://people.mozilla.org/~gavin/bug/235853/

The two "diff" files represent the only significant difference between the two logs that I could find - in the "bug" case, two nsHTTPChannels are created, while in the "no bug" case, only one is.

I also got a stack to the code that prompts about the form resubmit:
http://people.mozilla.org/~gavin/bug/235853/repost.alert.stack
Comment 93 :Gavin Sharp [email: gavin@gavinsharp.com] 2008-03-25 23:41:08 PDT
Created attachment 311742 [details]
HTTP logs/testcase/stack

This contains the current contents of http://people.mozilla.org/~gavin/bug/235853/, for posterity.
Comment 94 georgi - hopefully not receiving bugspam 2008-03-25 23:44:07 PDT
(In reply to comment #88)
> Do you think that the
> DNS is changing between resolution points?
> 

yes, malicious dns server on purpose gives different replies for the ip of
hostname with short dns TTL. so if a script is loading several times content
from mal.dns you may end up with content with hostname mal.dns coming from two
ip-s: 1.1.1.1 and 10.1.1.1 and both seem coming from the same hostname so they
can interact.
Comment 95 Boris Zbarsky [:bz] 2008-03-26 07:16:06 PDT
Gavin, can you load the POST result, then breakpoing in CanSavePresention and navigate away from it?  Does it get bfcached?  If not, which exact reason causes CanSavePresentation to return false?  Is doing loads directly from the onload handler relevant?
Comment 96 :Gavin Sharp [email: gavin@gavinsharp.com] 2008-03-26 09:45:56 PDT
(In reply to comment #95)
> Gavin, can you load the POST result, then breakpoing in CanSavePresention and
> navigate away from it?  Does it get bfcached?  If not, which exact reason
> causes CanSavePresentation to return false?

With or without the patch applied (i.e. whether or not the dialog appears), the CanSavePresentation called with nsLocation::SetHref on the stack (i.e. the one that navigates away from the POST in the testcase) returns false due to the subframe check:

http://bonsai.mozilla.org/cvsblame.cgi?file=mozilla/docshell/base/nsDocShell.cpp&rev=1.894&mark=5292#5261

> Is doing loads directly from the onload handler relevant?

No - I can reproduce the problem by driving the test manually (e.g. submitting the iframe's form with a javascript: URI, and then just going back to the form page, and forward to the POST result manually with the back/forward buttons).
Comment 97 Boris Zbarsky [:bz] 2008-03-26 09:57:45 PDT
Oh, right.  The whole point was to not have bfcache happening so the loads have to come from cache...

I guess it would be worth stepping through the HTTP channel AsyncOpen/OpenCacheEntry stuff when returning to the POST page and see what's up (nothing in cache, cache entry invalid, or what).
Comment 98 :Gavin Sharp [email: gavin@gavinsharp.com] 2008-03-26 19:53:14 PDT
Created attachment 311954 [details] [diff] [review]
regression fix?

OK, I looked into this a bit further. When the bug occurs, nsHttpChannel::OpenCacheEntry() fails when going back to the POSTed page. If fails because it has the wrong mPostID, so it ends up generating the wrong cacheKey to pass to nsCacheSession::OpenCacheEntry.

I added some debug logging and noticed that docshell was setting the new cachekey on the channel, but that it wasn't setting it on the right channel when the bug occurred ( http://people.mozilla.org/~gavin/bug/235853/cachekey_comparison.txt is a log of the relevant OpenCacheEntry/SetCacheKey calls). That made me think of comment 92, where I noticed multiple channels being created. That led to finding SetupReplacementChannel(), where I noticed that mPostID wasn't being transferred to the new channel. I made it do that, and then I could no longer reproduce the bug.

I'm still not sure why shaver's patch causes the extra channel to be present, and I have no idea whether this change is correct in the general case, but it fixes the regression for me.
Comment 99 Boris Zbarsky [:bz] 2008-03-26 22:40:10 PDT
You should probably only be sending the mPostId along if the POST data is being sent along too, right?  And possibly only if the URI hasn't changed... not sure about that last.  biesi might know.

> I'm still not sure why shaver's patch causes the extra channel to be present,

Coming from DoReplaceWithProxy, right?
Comment 100 Christian :Biesinger (don't email me, ping me on IRC) 2008-03-27 14:29:35 PDT
Ah right. I should've thought of this.

The patch looks basically correct to me, except that the cache key should only be forwarded when coming from ReplaceWithProxy, I would say. maybe add an argument forwardCacheKey to SetupReplacementChannel.
Comment 101 :Gavin Sharp [email: gavin@gavinsharp.com] 2008-03-27 17:27:25 PDT
Created attachment 312162 [details] [diff] [review]
regression fix
Comment 102 :Gavin Sharp [email: gavin@gavinsharp.com] 2008-03-28 15:26:26 PDT
Created attachment 312383 [details] [diff] [review]
combined patch
Comment 103 Kelly Campbell 2008-03-31 06:16:08 PDT
*** Bug 425183 has been marked as a duplicate of this bug. ***
Comment 104 Mike Shaver (:shaver -- probably not reading bugmail closely) 2008-04-07 12:59:19 PDT
Comment on attachment 312383 [details] [diff] [review]
combined patch

Looking for approval here.  There is risk in bizarre PAC configurations, but the upside is pretty substantial in basically all PAC configurations.
Comment 105 Mike Beltzner [:beltzner, not reading bugmail] 2008-04-07 14:24:52 PDT
Comment on attachment 312383 [details] [diff] [review]
combined patch

a1.9=beltzner
Comment 106 :Gavin Sharp [email: gavin@gavinsharp.com] 2008-04-08 14:30:38 PDT
Checked this in again, and it broke some mochitests again - this time the offline cache ones. I could reproduce that bustage too, when I ran the offline tests directly. After spending some time trying to figure out what was going wrong in offline cache code, I noticed nsOfflineCacheUpdateItem::OpenChannel sets nsICachingChannel attributes on the channel - cacheClientID and cacheForOfflineUse - that weren't being forwarded to the new channel in nsHttpChannel::SetupReplacementChannel. So essentially the same problem that was exposed for cacheKey by the POST submission test.

Fixing SetupReplacementChannel to also forward those attributes to the new channel fixed that failure, but that makes me wonder whether there are other properties of the channel that need forwarding that might not have test coverage. cacheToken or offlineCacheToken, perhaps?
Comment 107 :Gavin Sharp [email: gavin@gavinsharp.com] 2008-04-08 14:39:51 PDT
Oh, I guess the setters for those aren't implemented, so probably not?
Comment 108 :Gavin Sharp [email: gavin@gavinsharp.com] 2008-04-08 14:47:46 PDT
Created attachment 314416 [details] [diff] [review]
combined patch + additional fix

I'm running this through mochis again.
Comment 109 :Gavin Sharp [email: gavin@gavinsharp.com] 2008-04-08 15:46:41 PDT
I'm seeing another failure now:
source/extensions/cookie/test/test_loadflags.html is failing, because its "http-on-modify-request" observer (in:
http://lxr.mozilla.org/seamonkey/source/extensions/cookie/test/file_testloadflags.js )

is expecting to be called twice, but is now being called 4 times. This is presumably another consequnce of two channels being used per load rather than one. It seems to me that the test should probably be adjusted to now expect 4 calls, but I'm not sure.
Comment 110 dwitte@gmail.com 2008-04-08 20:18:16 PDT
yeah, i suppose adjusting it to 4 is fine - it's (i hope!) an implementation detail as to how many requests are generated for a load. though biesi should probably confirm that this behavior is okay, and not bad in other ways.

a more robust way to write the test might be to pull the request URI from the channel on each firing of "http-on-modify-request", and compare it against an expected list. but r=me for making it 4, as long as you add a comment in the .js explaining why we expect twice the calls.
Comment 111 Christian :Biesinger (don't email me, ping me on IRC) 2008-04-09 08:07:04 PDT
yes, the number of channels used is an implementation detail. that's why nsIChannelEventSink has an _INTERNAL reason.
Comment 112 Mike Shaver (:shaver -- probably not reading bugmail closely) 2008-04-14 07:51:07 PDT
Comment on attachment 314416 [details] [diff] [review]
combined patch + additional fix

biesi: can you stamp the combined change?  We'll get the mochitest count before checkin as well, per dwitte.
Comment 113 :Gavin Sharp [email: gavin@gavinsharp.com] 2008-04-16 21:07:23 PDT
Created attachment 316154 [details] [diff] [review]
rollup patch

Includes the change to the cookie unit test. Passes a full run of Mochitest on a fresh build.
Comment 114 Mike Beltzner [:beltzner, not reading bugmail] 2008-04-16 21:32:42 PDT
Comment on attachment 316154 [details] [diff] [review]
rollup patch

a1.9=beltzner
Comment 115 :Gavin Sharp [email: gavin@gavinsharp.com] 2008-04-17 13:34:15 PDT
Been waiting for the tree to go green so that I can check this in.
Comment 116 :Gavin Sharp [email: gavin@gavinsharp.com] 2008-04-17 20:21:30 PDT
After all the work I'd done to make absolutely sure that this passed a full run of mochitests, multiple times, I checked it in. Then I discovered that it caused one of the mochi*chrome* tests to fail:

http://lxr.mozilla.org/seamonkey/source/content/base/test/chrome/test_bug421622.xul

I don't know why, offhand - SetupReplacementChannel certainly does transfer the referer.

I'm very upset that I had to back this out *again* because of my own negligence. At least we know we have good test coverage of PAC-enabled network code paths, I guess.
Comment 117 Peter da Silva 2008-04-28 09:40:43 PDT
I get continual hangs even if I'm not doing anything. I assumed that this was caused by Camino 1.6, but I backed out to Camino 1.5.5 and I'm still getting it... so I went in with Activity Monitor and found this was always due to a bunch of nested js_Invoke called by nsPACMan::getProxyForURI, called, at the top level, from -[Bookmark refreshIcon]

I'm not sure that this is due to the problem being discussed here.

A couple of questions are pulled irresistibly to mind here.

1. What is -[Bookmark refreshIcon] being called so frequently for?

2. Shouldn't ANY kind of name resolution related work (including proxies and PAC files) happen in its own thread (or at least be interruptible)? Or even in a whole separate proxy helper process - it's not like name lookup is fast enough that even a few *milli*seconds to pass a name to the helper and get the address back would be noticeable.

3. Speaking of helper applications, what about putting all the maintenance work like updating bookmark icons into one?
Comment 118 Peter da Silva 2008-04-28 09:43:05 PDT
Created attachment 318169 [details]
Activity monitor samples during a typical "proxy hang".

Just for information, the Activity Monitor samples from a typical "proxy hang" in Camino.
Comment 119 Mike Shaver (:shaver -- probably not reading bugmail closely) 2008-04-28 10:00:14 PDT
The hangs under GetProxyForURI are due to exactly this bug.  The helper app and bookmark icon bits should be in other Camino bugs, and probably not mentioned here again.

DNS resolution does happen on a background thread, but proxy selection is currently synchronous, so the script waits for the background thread to have completed, making the asynchronicity of the actual resolution somewhat pointless.

What we need to fix this is for someone to help figure out what is causing the mochichrome failure, and then we can look at how to get a patch like this into a 3.0.1.  The motivating problem is well-understood, and easy to reproduce.
Comment 120 Stuart Morgan 2008-04-28 11:58:14 PDT
(In reply to comment #119)
> The helper app and bookmark icon bits should be in other Camino bugs

Specifically, bug 351678.
Comment 121 Mike Schroepfer 2008-05-09 14:08:28 PDT
Comment on attachment 316154 [details] [diff] [review]
rollup patch

Removing approval since this missed the 1.9 cutoff.
Comment 122 Mubeen 2008-07-02 07:10:15 PDT
I am using Firefox in an enterprise environment. This bug has been made me switch to opera. Can anyone tell whether this bug is going to be fixed in the next update for Firefox 3.0.
Comment 123 Jo Hermans 2008-07-02 07:24:21 PDT
(In reply to comment #122)
> I am using Firefox in an enterprise environment. This bug has been made me
> switch to opera. Can anyone tell whether this bug is going to be fixed in the
> next update for Firefox 3.0.
> 

see comment 119 : there's still a testcase that is failing, and that needs to be fixed first
Comment 124 Jeff Muizelaar [:jrmuizel] 2008-07-09 07:45:20 PDT
(In reply to comment #116)
> After all the work I'd done to make absolutely sure that this passed a full run
> of mochitests, multiple times, I checked it in. Then I discovered that it
> caused one of the mochi*chrome* tests to fail:
> 
> http://lxr.mozilla.org/seamonkey/source/content/base/test/chrome/test_bug421622.xul
> 
> I don't know why, offhand - SetupReplacementChannel certainly does transfer the
> referer.

The problem appears to be that test_bug421622.xul uses setRequestHeader to set the referer and so nsHttpChannel::mReferrer is not set and thus not copied over by SetupReplacementChannel.

I've attached a patch that adds a hack to make the mochichrome test pass, but also adds a new mochitest that uses a custom header that is not propagated by SetupReplacementChannel.

I'd guess that both test_bug421622.xul and the new test would currently fail if a proxy is setup so that DoReplaceWithProxy...SetupReplacementChannel etc. is called like it is after this patch is applied. However, I haven't confirmed this.
Comment 125 Jeff Muizelaar [:jrmuizel] 2008-07-09 07:48:39 PDT
Created attachment 328677 [details] [diff] [review]
rollup patch + hack to set referer header

Copies the referer header over to make the mochichrome test pass. Includes a new test that still fails.
Comment 126 Boris Zbarsky [:bz] 2008-07-09 08:03:49 PDT
Hmm.  So this patch as it stands doesn't carry over "custom" headers set by XMLHttpRequest?  That seems like it could be a problem, no?
Comment 127 Jeff Muizelaar [:jrmuizel] 2008-07-09 08:16:07 PDT
(In reply to comment #126)
> Hmm.  So this patch as it stands doesn't carry over "custom" headers set by
> XMLHttpRequest?  That seems like it could be a problem, no?
> 

Correct. The patch is not meant to be applied. However, SetupReplacementChannel already has the problem. The patch from shaver/gavin just makes the problem worse because we are calling SetupReplacementChannel more often.
Comment 128 Boris Zbarsky [:bz] 2008-07-09 08:22:49 PDT
Right.  I understand where the patches stand.

It seems that SetupReplacementChannel is used for two things: 3xx redirects and proxies.  I'm not sure whether copying headers en masse makes sense for these two use cases.  The HTTP RFC seems to be silent on the behavior that 3xx should lead to in this regard...
Comment 129 Jeff Muizelaar [:jrmuizel] 2008-07-09 14:06:21 PDT
Created attachment 328746 [details] [diff] [review]
rollup patch + copy all headers

This patch copies all of the headers in the case of a proxy connection and no headers otherwise. It renames the previous 'transferCacheInfo' flag to 'forProxy' (I don't like this name, but nothing better came to mind).

It passes the test_bug421622.xul test as well as the newly added test.
Comment 130 Christian :Biesinger (don't email me, ping me on IRC) 2008-07-29 17:08:14 PDT
Comment on attachment 328746 [details] [diff] [review]
rollup patch + copy all headers

This will cause kind of interesting behaviour because now for some
OnChannelRedirect calls we won't have transferred the headers while
for others (REDIRECT_INTERNAL) we will have. I guess that's not that
much of a problem since the observer will probably just re-set the
header if it cares at all.

On the patch itself:
      for (i = 0; i < count; ++i) {

The body of this loop is incorrectly indented. It seems you're using tabs,
please don't.

	nsCAutoString header, value;
	header.Assign(oldHeader);
	value.Assign(oldValue);
	httpChannel->SetRequestHeader(header, value, PR_FALSE);

This should just be:
  httpChannel->SetRequestHeader(nsDependentCString(header),
                                nsDependentCString(value),
                                PR_FALSE);

in the header:
    nsresult SetupReplacementChannel(nsIURI *, nsIChannel *, PRBool preserveMethod, PRBool forProxy);

Limit that line to 80 characters
Comment 131 Jeff Muizelaar [:jrmuizel] 2008-08-05 10:51:09 PDT
Created attachment 332389 [details] [diff] [review]
breaks lines netwerk/protocol/http/src/nsHttpChannel.h

Breaks lines longer than 80 columns in netwerk/protocol/http/src/nsHttpChannel.h
Comment 132 Jeff Muizelaar [:jrmuizel] 2008-08-05 10:56:31 PDT
Created attachment 332390 [details] [diff] [review]
rollup patch + copy all headers

The previous patch with changes suggested by cbiesinger.
Comment 133 Ted Mielczarek [:ted.mielczarek] 2008-09-30 07:50:27 PDT
*** Bug 306922 has been marked as a duplicate of this bug. ***
Comment 134 Mike Shaver (:shaver -- probably not reading bugmail closely) 2008-09-30 08:13:16 PDT
Comment on attachment 332390 [details] [diff] [review]
rollup patch + copy all headers

Moving sr to bz, because biesi was the r, I believe.  Maybe we'll get lucky for 3.1b1!
Comment 135 Mike Shaver (:shaver -- probably not reading bugmail closely) 2008-09-30 08:47:10 PDT
Comment on attachment 332390 [details] [diff] [review]
rollup patch + copy all headers

I'm a maroon -- r+sr was from biesi on previous patches, restoring request.
Comment 136 Christian :Biesinger (don't email me, ping me on IRC) 2008-09-30 09:49:17 PDT
Comment on attachment 332390 [details] [diff] [review]
rollup patch + copy all headers

+      PRUint32 i, count = mRequestHead.Headers().Count();
+      for (i = 0; i < count; ++i) {

why not:
  for (PRUint32 i = 0; ...)

+        httpChannel->SetRequestHeader(nsDepedentCString(header), nsDependentCString(value), PR_FALSE);

please limit your lines to 80 chars

+        httpChannel->SetRequestHeader(nsDepedentCString(header), nsDependentCString(value), PR_FALSE);

you misspelled nsDependentCString in the first argument?
Comment 137 Christian :Biesinger (don't email me, ping me on IRC) 2008-09-30 09:50:20 PDT
Also, a necko unit test instead of just the one in content/ would be really great.
Comment 138 Jeff Muizelaar [:jrmuizel] 2008-09-30 15:06:33 PDT
Created attachment 341161 [details] [diff] [review]
v2

This addresses the review comments and merges with HEAD. 

Part of the merge involved a new call to SetupReplacementChannel() in ProcessFallback(). I pass in false as 'forProxy', but this is just a guess as to the desired behavior. Does it seem correct?
Comment 139 Christian :Biesinger (don't email me, ping me on IRC) 2008-10-06 10:27:21 PDT
(In reply to comment #138)
> Part of the merge involved a new call to SetupReplacementChannel() in
> ProcessFallback(). I pass in false as 'forProxy', but this is just a guess as
> to the desired behavior. Does it seem correct?

yeah that seems right.

are you going to add a test in netwerk/test/unit? you'd just have to set the PAC pref using the prefservice, can be a data: URI that just always returns DIRECT I think. and then check that stuff gets correctly transferred.
Comment 140 Jeff Muizelaar [:jrmuizel] 2008-10-06 19:54:53 PDT
I'm not going to have time to write a test for this soon. I still can when I get the chance, but someone else could probably write a test much faster than I could anyways.
Comment 141 Benjamin Smedberg [:bsmedberg] 2008-10-29 09:27:30 PDT
Not going to block on this, but I think we want it even after beta2... what's the current status of the patches here?
Comment 142 Jeff Muizelaar [:jrmuizel] 2008-10-29 14:38:57 PDT
Created attachment 345365 [details] [diff] [review]
Necko unit test

Here's a necko unit test that tests that headers and cache keys are preserved when we use a proxy. It fails with trunk and passes with this patch.
Comment 143 Jeff Muizelaar [:jrmuizel] 2008-11-03 14:51:49 PST
Created attachment 346138 [details] [diff] [review]
[1/2] break lines in netwerk/protocol/http/src/nsHttpChannel.h
Comment 144 Jeff Muizelaar [:jrmuizel] 2008-11-03 14:57:30 PST
Created attachment 346141 [details] [diff] [review]
[2/2] v3: Defer proxy resolution for HTTP and HTTPS PAC to avoid blocking main thread during DNS resolution

Defers proxy resolution for HTTP and HTTPS PAC to avoid blocking main thread during DNS resolution and adds code to SetupReplacementChannel to ensure that the proxy channel is similar to the original channel.

It also adds a necko unit test and a mochi test to that the proxy channel preserves the needed properties of the original channel.
Comment 145 Jeff Muizelaar [:jrmuizel] 2008-11-04 16:50:26 PST
Created attachment 346370 [details] [diff] [review]
[2/2] v4: Defer proxy resolution for HTTP and HTTPS PAC to avoid blocking main thread during DNS resolution

Fix a grammar bug in v3
Comment 146 :Gavin Sharp [email: gavin@gavinsharp.com] 2008-11-04 21:52:16 PST
This still isn't passing mochitest :( It's causing tests/content/base/test/test_CrossSiteXHR.html to fail now.
Comment 147 Boris Zbarsky [:bz] 2008-11-05 10:19:20 PST
The most immediate problem is that nsCrossSiteListenerProxy::OnChannelRedirect is broken: it never allows the redirect, because it calls CheckRequestApproved which always fails on channels that are not 2xx HTTP responses (which no channel being redirected ever is).

Sicking says fixing that won't be enough because preflight requests (for POST/PUT) aren't allowed to be redirected right now; we'd need to fix that to allow the PAC redirect for those.
Comment 148 Peter da Silva 2008-11-07 04:12:33 PST
Preflight requests? Are those the ones where the browser preloads links? It would be worth disabling them for proxied connections, perhaps?
Comment 149 Boris Zbarsky [:bz] 2008-11-07 05:21:47 PST
Preflight requests are the access-control GET requests made before a cross-site XHR POST is allowed to happen.
Comment 150 Jeff Muizelaar [:jrmuizel] 2008-11-13 13:23:54 PST
I was talking to Boris and he suggested that Jonas disable the failing tests so that we can land this in time to make the beta. The reason the tests are failing comes from this patch causing existing bugs in the tree to be exercised and the benefit from the additional user testing probably outweighs the disadvantage of having the disabled tests.
Comment 151 Jonas Sicking (:sicking) No longer reading bugmail consistently 2008-11-13 14:00:26 PST
I'm fine with disabling the tests for now I guess. Though it will mean that XSXHR won't work for people with proxies at all for now :(
Comment 152 Jeff Muizelaar [:jrmuizel] 2008-11-13 14:04:15 PST
What tests need to be disabled?
Comment 153 Boris Zbarsky [:bz] 2008-11-13 14:08:36 PST
Per comment 146, content/base/test/test_CrossSiteXHR.html 

Just commenting it out in the Makefile should work...

Jonas, did you ever get that patch I sent you into a bug?
Comment 154 Jeff Muizelaar [:jrmuizel] 2008-11-13 14:13:13 PST
test_XHRDocURI.html was also hanging.
Comment 155 Jeff Muizelaar [:jrmuizel] 2008-11-13 15:07:16 PST
test_CrossSiteXHR_cache.html also breaks.
Comment 156 Boris Zbarsky [:bz] 2008-11-13 18:06:03 PST
Comment on attachment 346370 [details] [diff] [review]
[2/2] v4: Defer proxy resolution for HTTP and HTTPS PAC to avoid blocking main thread during DNS resolution

If we're thinking about doing this in the beta....
Comment 157 Mike Beltzner [:beltzner, not reading bugmail] 2008-11-14 05:51:53 PST
Comment on attachment 346370 [details] [diff] [review]
[2/2] v4: Defer proxy resolution for HTTP and HTTPS PAC to avoid blocking main thread during DNS resolution

I'm fine to take this opportunistically if people tell me that it's well tested and won't cause headaches, but based on the previous comments, I won't take this without a patch that also disables the tests that need to be disabled, and without a follow-up bug filed to fix those tests.
Comment 158 Boris Zbarsky [:bz] 2008-11-14 06:42:29 PST
> Though it will mean that XSXHR won't work for people with proxies at all for now

It already doesn't in a number of cases (proxy failover, say).

Jeff, can you put together a patch that does the test disabling?  I'll work on getting that bug filed.
Comment 159 Jeff Muizelaar [:jrmuizel] 2008-11-14 10:53:58 PST
So it looks like we also fail a bunch of video tests:
 /tests/content/media/video/test/test_constants.html
 /tests/content/media/video/test/test_bug461281.html  - Bug 461281
 /tests/content/media/video/test/test_duration1.html
 /tests/content/media/video/test/test_ended1.html

basically it looks like video is also broken with proxies with this patch.
Comment 160 Boris Zbarsky [:bz] 2008-11-14 11:29:42 PST
I filed bug 464954 on fixing XHR.

Is the problem that video is using the same access-control stuff as XHR (and hence is broken in the same ways)?
Comment 161 Boris Zbarsky [:bz] 2008-11-14 11:35:52 PST
Actually, video doesn't use access-control yet.  Chris, any idea what's going on there?  Does video work across 3xx redirects?
Comment 162 cajbir (:cajbir) 2008-11-14 13:38:12 PST
I think Chris Pearce worked on some things to do with redirects on bug 451958. Hopefully he can comment on whether redirects require that patch or not.
Comment 163 Chris Pearce (:cpearce) 2008-11-16 13:49:36 PST
(In reply to comment #161)
> Does video work across 3xx redirects?

It WFM in recent win32 nightlies. See: http://pearce.org.nz/movie
That's a 301 redirect.

Also the tests for bug 451958 test redirects with allow-origin and non-allow-origin videos, and that WFM. That patch uses nsCrossSiteListenerProxy... That patch hasn't landed yet, it's blocked by bug 462878, which I'm working on now. 


(In reply to comment #159)
> So it looks like we also fail a bunch of video tests:
>  /tests/content/media/video/test/test_constants.html
> basically it looks like video is also broken with proxies with this patch.

test_constants.html doesn't load any video, it only tests the existence and value of constants. I don't understand the area you're working in, but I can't imagine how anything to do with networking could affect this testcase.

Maybe try turning tracing off? We've been getting random test failures with video, maybe that's the cause?
Comment 164 Jeff Muizelaar [:jrmuizel] 2008-11-18 13:42:06 PST
test_constants.html only fails when being run after test_bug461281.html

test_bug461281.html fails consistently with:
not ok - Error thrown during test: [object ProgressEvent] got 0, expected 1

If you give the patch the PAC patch a try you should be able to reproduce the test failures without any difficulty and it's not unlikely that they point to bugs in the video code.
Comment 165 Tyler Downer [:Tyler] 2008-11-28 17:30:10 PST
*** Bug 450153 has been marked as a duplicate of this bug. ***
Comment 166 lsof 2009-01-16 08:39:07 PST
This bug turns a mistyped key to become a minute of annoyance, it will be good when it is fixed :)
Comment 167 Jeff Muizelaar [:jrmuizel] 2009-01-17 09:19:12 PST
I reran the media/video tests with a recent tree and they all seem to pass. So it looks like we're good there.

I also tried on the patch from bug 464954, but it seems to cause content/base/test/test_CrossSiteXHR.html to hang, event without the PAC patch.
Comment 168 Jonas Sicking (:sicking) No longer reading bugmail consistently 2009-01-17 09:45:52 PST
I'll land bug 464954 today when I get back from the ski slopes, that should help with the cross-site XHR tests.
Comment 169 Jeff Muizelaar [:jrmuizel] 2009-01-23 06:37:20 PST
It looks like content/html/document/test/test_bug445004.html fails with this patch.
Comment 170 Boris Zbarsky [:bz] 2009-01-23 06:50:56 PST
That's quite odd.  Which part of that test?
Comment 171 Jeff Muizelaar [:jrmuizel] 2009-01-23 06:56:12 PST
I'm not sure. I wasn't able to reproduce the failure when running the test individually. I'm rerunning the whole suite to see if I can reliably reproduce the failure that way.
Comment 172 Jeff Muizelaar [:jrmuizel] 2009-01-23 07:21:19 PST
So, I wasn't able to reproduce the test_bug445004.html failure but toolkit/components/places/tests/test_bug_411966.html seems to fail reliably with:

"not ok - Got wrong referrer for http://localhost:8888/tests/toolkit/components/places/tests/bug_411966/PermRedirectPage.htm"

and toolkit/components/url-classifier/tests/mochitest/test_classifier.html and toolkit/components/url-classifier/tests/mochitest/test_classifier_worker.html both fail with:

"not ok - Error thrown during test: document.getElementById("testFrame") is null got 0, expected 1"
Comment 173 Jeff Muizelaar [:jrmuizel] 2009-01-23 08:32:50 PST
test_classifier.html and test_classifier_worker.html
still fail without the patch so I don't know what's up there.

test_bug_411966.html passes without the patch and the redirect stuff seems like it could be related to the patch.
Comment 174 Boris Zbarsky [:bz] 2009-01-23 08:46:13 PST
Could be, yes... That said, you do copy the referrer in SetupReplacementChannel, right?

Can you reproduce the failure when running just the one test?  If so, are you willing to debug?
Comment 175 Jeff Muizelaar [:jrmuizel] 2009-01-23 08:49:31 PST
Here's some more information about how the test fails:

not ok - Got wrong referrer for http://localhost:8888/tests/toolkit/components/places/tests/bug_411966/PermRedirectPage.htm, expected http://localhost:8888/tests/toolkit/components/places/tests/bug_411966/ClickedPage.htm 

The error happens in redirect.js:checkDBOnTimeout():181
Comment 176 Jeff Muizelaar [:jrmuizel] 2009-01-23 08:57:39 PST
We copy the Referrer if mReferrer is set and we copy all the headers so it should be getting set.

I can reproduce the failure when running just the one test and I'm willing to debug with guidance.
Comment 177 Boris Zbarsky [:bz] 2009-01-23 09:05:19 PST
Hmm. It looks like places stores this redirect with addDocumentRedirect.  If you skip doing that one call for cases when the redirect is internal, does that help?
Comment 178 Jeff Muizelaar [:jrmuizel] 2009-01-23 12:29:03 PST
It does help. Not calling addDocumentRedirect() when the redirect is internal fixes the test case.

So now, to fix it properly:

1. AddDocumentRedirect() doesn't check for internal redirects. It seems reasonable that it should ignore internal redirects.

2. nsDocShell::OnRedirectStateChange() also calls AddDocumentRedirect() and gavin was wondering whether nsDocShell::OnRedirectStateChange() should be called on internal redirects.
Comment 179 Boris Zbarsky [:bz] 2009-01-26 10:53:35 PST
For #2, I would say yes.  For example, the URL classifier probably needs to see the new channel, as far as I can tell.
Comment 180 :Gavin Sharp [email: gavin@gavinsharp.com] 2009-01-26 11:09:30 PST
(In reply to comment #179)
> For #2, I would say yes.  For example, the URL classifier probably needs to see
> the new channel, as far as I can tell.

Right, but can that not be done in nsDocLoader::OnChannelRedirect? I guess I don't really understand why we need both nsDocShell::OnRedirectStateChange and nsDocLoader::OnChannelRedirect, if the difference wasn't meant to be that only the former was called for non-INTERNAL redirects. Not really relevant to this bug anymore, I guess...
Comment 181 Jeff Muizelaar [:jrmuizel] 2009-01-29 06:33:45 PST
After some more testing I'm getting occasional failures on test_bug445004.html, however I haven't been able to reliably reproduce the problem.
Comment 182 Jeff Muizelaar [:jrmuizel] 2009-03-03 11:47:42 PST
I can reliably reproduce the following failure with tryserver unit tests:
*** 41223 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/geolocation/test_manyWindows.html | Error thrown during test: windows[i] is null - got 0, expected 1

There also seem to be some leak test issues:
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1236092965.1236101757.29727.gz&fulltext=1
Comment 183 Jeff Muizelaar [:jrmuizel] 2009-03-03 11:59:59 PST
Doug, can you give some guidance as to what could be going wrong here?
Comment 184 John Vivirito 2009-03-22 12:14:34 PDT
seems to be same as bug 208287 and should be marked a dup of this bug since there is more info on this bug than other.
Comment 185 Jeff Muizelaar [:jrmuizel] 2009-05-04 18:30:31 PDT
Jason can you take ownership of this bug?
Comment 186 Wayne Mery (:wsmwk, NI for questions) 2009-05-22 06:09:24 PDT
*** Bug 483544 has been marked as a duplicate of this bug. ***
Comment 187 Jason Duell [:jduell] (needinfo me) 2009-05-26 12:24:25 PDT
> Jason can you take ownership of this bug?

Sure.   I've got a lot on my plate, but probably not as much as Shaver (I didn't even know he fixed bugs...)

I haven't had time to look over the 186 comments, but the first thing that comes to mind is that multi-process (electrolysis) may make this bug moot, assuming that DNS lookup occurs in the subprocess and not in the chrome process.  Does anyone have a sense if that is correct?   If so, unless we're very close to a architecturally clean fix here, it may make sense to wait and get this "for free" and "right way" later.
Comment 188 Jason Duell [:jduell] (needinfo me) 2009-05-26 12:28:08 PDT
Sorry, upon 2 seconds more reflection, it's unlikely that electrolysis is going to automatically fix this for free (though it may change how we architect the fix, if we can keep the main thread uninvolved or less involved).
Comment 189 Daniel Einspanjer [:dre] [:deinspanjer] 2009-05-28 09:24:42 PDT
I'm trying to do some research and dupe checking on bug 492558 because I saw three separate reports of the problem as being new to Firefox 3.0.10.  The technical aspects of it are a little over my head, so I wanted to ask if it might be related in any way to this bug since one is receiving current attention.
Comment 190 RNicoletto 2009-07-21 08:51:31 PDT
With the new version of Firefox (Mozilla/5.0 (Windows;
U; Windows NT 5.1; fr; rv:1.9.1.1) Gecko/20090715 Firefox/3.5.1), it seems
there are no more freeze.

Anyone can confirm this?
Comment 191 Steven 2009-07-21 08:54:48 PDT
No, I still experience freezes just as before.

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.1) Gecko/20090715 Firefox/3.5.1
Comment 192 Ben 2009-07-21 09:02:04 PDT
I also continue having problems:

Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9.1.1) Gecko/20090715 Firefox/3.5.1 (.NET CLR 3.5.30729)
Comment 193 Henrik Skupin (:whimboo) 2009-08-24 04:53:43 PDT
*** Bug 489772 has been marked as a duplicate of this bug. ***
Comment 194 Kosta 2009-11-19 05:31:08 PST
I DON'T use PAC. I use manual proxy configuration.
I have the same problem of blocking UI during opening a new tab/URL.
I don't understand for what reason DNS resolving has to be done in case using PROXY.

However I understand that this DNS resolving is done through the Proxy server!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
I have ping google.com and it works fine for any server in the world.
I need just one simple property in the about:config or checkbox in the proxy settings that says:

Do the DNS resolving with direct connection.

By using this simple fix i will have Firefox that resolves names using the local caching DNS server and working as fast as possible.
Please do that before I download the source and make complete mess of the Firefox to make it works that way.

This proposal will not help for the half of the people that have only proxy and local DNS server.
However many companies including big ones like the one I work for, uses proxy for internet and direct communication for VPN connection to all the offices worldwide. And this is not going to change any time soon in this millennium.
I suffer this bug from the time when Firebird 0.4 have been released. I use Firefox 3.5.4 WinXP and it is not fixed YET WHY?
Comment 195 Jo Hermans 2009-11-19 06:03:11 PST
DNS resolving *has* to be done in the proxy server, otherwise it wouldn't know where to forward the request to.

A direct connection does not use a proxy server, so the browser has to do all the DNS lookups itself.

In the case of a PAC file, your browser might be doing DNS lookups too, but not always (depending on the content of the PAC file).

You're probably confusing the DNS prefetching-lookups (sent by the browser) with the ones needed to open the connection (sent by the browser and/or the proxyserver). In case of a PAC file, they're not always necessary, so we might disable prefetching in that case. See bug 507578.
Comment 196 Kosta 2009-11-25 04:18:41 PST
I do not know the PAC file scenario of the problem.
But seems my bug reported was redirected to this one. I suppose that the source code forced the maintainers of the bugs to do that.

I also wish to mention that the line in my previous comment was missing one word that changes the meaning. 
It was :

"This proposal will not help for the half of the people that have only proxy and
local DNS server."

It should be:

This proposal will not help for the half of the people that have only proxy and
NO local DNS server.
Comment 197 The Sent 2009-12-02 15:50:47 PST
Going back to an older post, this does seem to fix the issue:

function FindProxyForURL(url, host)
{
  //So the error message "no such host" will appear through the
  //normal Netscape box - less support queries :)
  if (!isResolvable(host))
    return "DIRECT";
...
Comment 198 Kosta 2009-12-10 06:50:47 PST
I have experienced the same hanging even not using any proxy (at home).
It seems to me that if the web server is on high load or the internet connection is slow, opening a new tab hangs until response. So I had no way to use other tabs until the last opened is loaded and least at 10% or so.
I will quite probably create a new bug about that if it is not already opened.
Comment 199 Woody Randolph 2010-01-10 13:25:36 PST
Perhaps I am in the wrong place, but I will give it a try.  For at least the next two months, I will be using this computer, which is now seven years old.  I am currently running Win 7 and not experiencing any problems that I am aware of at this time.  My ISP, Wild Blue, suggested that to improve the speed of my internet access, to use Firefox.  I have been using Internet Explorer since the beginning of my computer experience.  I was not satisfied and a little unhappy with IE 8, so I readily switched to FireFox.  The problem I have experienced, and continue to experience, occurs when I open a new tab or click on a link. It looks asa if it is going to do the requested action.  After several seconds, the progress bar has no indication that it has beun to load the page, or it may show progress then stop.  After a long period of time, a minute or more, I can "Stop" the procedure and then select the same thing again and it usually will quickly respond with the request.  In my settings, I am using an Auto-configuration proxy URL supplied by my ISP.  Would it be better to "Auto detect" the settings?  This is quite frustrating.
Comment 200 Jo Hermans 2010-01-10 14:37:22 PST
"Auto detect" is nothing magical, it's just a way for the browser to find out where the Automatic proxy configuration URL is located. In your case (if you're using wildblue.net as your DNS suffix), according to <http://help.wildblue.net/kb/article/2751>, I guess it will automatically look for the URL <http://wpad.wildblue.net/wpad.dat> instead of the <http://wpad.wildblue.com/wpad.dat> installed by the Wildblue Optimizer. I can't see from the outside if this is the same or not. If your DNS suffix is wildblue.com, then the 2 settings are exactly the same.

Anyway, there might be a problem with the proxy server(s) itself, or with the DNS request that your browser is using when using the wpad-file. Or with the pipelining that the Optimizer has installed (in which case it might also dependent on the exact website that you're visiting).

I'm not sure if you really need that wpad-file, it might be possible that you can surf faster without it. Or maybe you can try switching off the pipe-lining, especially the non-proxy pipelining, see <http://kb.mozillazine.org/Network.http.pipelining>.
Comment 201 jarod 2010-07-29 05:56:53 PDT
Kosta,
I have the same problem: I use manual proxy configuration and I get the UI freezes. So I had to ditch FF and use ie SIX !!!
Comment 202 Aragon Gouveia 2011-04-08 10:27:05 PDT
Is it just me, or has this bug gone from bad to horrible in version 4.0?  I used to avoid DNS related functions when writing PAC files, but firefox 4 seems to have intermittent freezing with even basic PAC files now.
Comment 203 Marcin Wisnicki 2011-04-15 08:08:16 PDT
It is indeed horrible!

Today I've tried accessing blender.org whose name servers do not respond.
When using PAC whole browser locks up for ~15s, unfreezes then locks up again.

Switching proxy to manual or direct resolves issue, though browser seems to freeze for almost second every 10-15 seconds while another tab tries to load page for ~1min.

This is on Win7 with FF4.0

I guess this can be reproduced by setting up a dns zone with ns records pointing to dead servers (or finding such domain on internet).
Comment 204 Marcin Wisnicki 2011-04-15 08:21:21 PDT
My wpad.dat is like this:

function
FindProxyForURL(url, host)
{
        if (isPlainHostName(host))
                return "DIRECT";
        if (dnsDomainIs(host, "my.local.domain"))
                return "DIRECT";
        if (isInNet(host, "127.0.0.0", "255.255.255.0"))
                return "DIRECT";
        if (isResolvable(host))
                return "PROXY 192.168.0.1:8080; DIRECT";
        else
                return "DIRECT";
}
Comment 205 Jo Hermans 2011-04-25 03:00:28 PDT
*** Bug 652496 has been marked as a duplicate of this bug. ***
Comment 206 RalfG 2011-06-19 08:11:25 PDT
It was version 2.0 of Firefox back in 2008, when I confirmed this bug for the first time. Three and a half years later, I have to confirm this bug for Firefox 4.0, too.

Especially for schools and bigger companies that cannot set proxy settings on each single client - but are important multipliers for the usage of free software - this is sad.

In times of muliple CPU cores and perfect threading, there should be no need for freezing GUIs, I think.

Now, I have to check if this is still a show stopper for the version 5.0 of Firefox. 

With kind regards.
RalfG (Skolelinux)
Comment 207 RalfG 2011-06-19 08:33:37 PDT
BTW, my wpad.dat looks like this (maybe it can be optimized?):
----
function FindProxyForURL(url, host)  {

//###############
//#Local Network#
//###############

  if (isInNet(host, "10.0.2.0", "255.255.254.0"))
  {
   return "DIRECT";
  }


//#########
//#Default#
//#########

return "PROXY 10.0.2.2:3128;"

}
Comment 208 Jo Hermans 2011-06-19 09:18:25 PDT
Yes, add this in front :

if (isPlainHostName(host) || shExpMatch(host, "*.yourdomain.com"))
{
   return "DIRECT";
}

If you're using internal webserver using the hostname or full DNS name, you will contact them directly, without going to the proxyserver first. The isInNet call only helps if you contact those webservers using an ipaddress, not a DNS-name.

But your problems have nothing to do with this bug, as your PAC-file does not use DNS at all (use dnsResolve call for that).
Comment 209 Emmanuel Colin 2011-06-19 10:17:01 PDT
I agree to Jo's solution for getting around the problem (I also added one set of rules with javascript regexps on the host variable to allow for direct use of IP adresses in the URL).

But I must disagree with his last comment, as it is my understanding that isInNet performs a dnsResolve call to get the IP adress in order to check it against the network mask.

Sad to hear that this bug is still live in 4.0...

Damn, now even Wikipedia says :
"The function dnsResolve (and similar other functions) performs a DNS lookup that can block your browser for a long time if the DNS server does not respond."
Comment 210 Scott Lamb 2011-06-19 12:29:01 PDT
Indeed, isInNet couldn't possibly work except by a DNS resolution. But the advice is good; best to avoid that call whenever possible.

Other workarounds: local proxy server: I wrote http://www.slamb.org/projects/foxigniter/ for this purpose back in 2008, reasoning that it was a lot easier to write than to get a patch into Firefox. (Given that the bug is around three years later, it appears I was right.) But I never finished it, so it has several problems including a crash when the local side of the proxy cancels a request and a lack of flow control without some libevent patches that probably won't apply cleanly to a modern version.

The reason I never finished it? Switched to Google Chrome instead. Maybe it's a bit rude of me to point this out on the Firefox bug tracker, but Chrome doesn't have this problem, so I'm much happier.
Comment 211 Bill Wood 2011-06-20 06:59:18 PDT
Firefox 4 seems worse.  When I restore a session of only 3 windows/20 tabs, it can take several minutes before the UI starts responding under Windows XP. This morning I killed the Firefox task after about 2 minutes and re-started it.  Firefox did not hang the second time I started it.  But periodically during the day it will hang for several minutes.
Comment 212 Henrik Skupin (:whimboo) 2011-06-20 07:10:15 PDT
Bill, your issue has absolutely nothing to do with this bug. If you can't find another one already filed please create a new one.
Comment 213 Bill Wood 2011-06-20 07:22:59 PDT
@Henrik, not sure why you say that.  My organization has a huge proxy.pac.  At home with no proxy, Firefox works fine with 20 windows and 50-100 tabs.
Comment 214 RalfG 2011-06-20 08:37:52 PDT
Hello & thanks for your comments.

Today I tested Firefox 5.0 (for Windows), and the bug is still there.

This is the way I test it:

(1) Put proxy settings to "auto detect"
(2) Start a web search for "test"
(3) Click on 3..4 hits of the search engine, using the mid-mouse button.

Usually, I won't get to the 3rd click, because Firefox hangs then.
Even if you open the Settings dialog before loading is complete, this
dialog will hang for several seconds.

If you use explicit proxy settings (10.0.2.2:3128), there is no delay at all.
Comment 215 RalfG 2011-06-20 08:43:11 PDT
When I get you right, auto proxy recognition can't do without a DNS call.
Reading this, I suggest:
- prevent DNS calls from blocking the GUI
- prevent DNS calls if the proxy string spread via DHCP is an IP (as in my case)
- prevent proxy detection with every URL call; rather define some update interval
  or just lookup proxy settings (wpad.dat) on failure.
Comment 216 GrahamM 2011-07-13 02:10:59 PDT
We had users complain about UI hang when they typed in a non-existant domain.  We had 30 IsInNet statements in the PAC file they were using.  Firing up Wireshark showed 2 types of lookup were attempted - normal DNS lookup, and a netbios lookup.  Regular DNS lookups returned pretty instantly with NXDOMAIN.

I saw the netbios lookup packets conformed to NameSrvQueryCount and NameSrvQueryTimeout defaults as mentioned here http://support.microsoft.com/kb/314053.  1.5seconds * 3 attempts = 4.5 seconds per IsInNet clause.

Completely turning netbios off fixed the issue, but that's clearly a no-go (not a Windows engineer, I'd have happily turned it off!).

Someone in our team came up with the idea of cutting down on the amount of resolving needed.

We went from the following:

=== from ===
if (isInNet(host, "172.31.0.0", "255.255.0.0") ||
    isInNet(host, "10.0.1.0", "255.255.255.0"))
    {
    return "proxy1.domain.co.uk:1234"
    }
    if (isInNet(host, "192.168.0.0", "255.255.0.0") ||
        isInNet(host, "127.0.0.0", "255.0.0.0") ||
        isInNet(host, "172.16.0.0", "255.255.0.0") ||
        isInNet(host, "172.19.4.0", "255.255.252.0") ||
        ....
        ....
        isInNet(host, "172.20.0.0", "255.255.0.0") ||
        isInNet(host, "172.21.0.0", "255.255.0.0") ||
        isInNet(host, "10.0.0.0", "255.0.0.0"))
    {
    return "DIRECT"
 
=== to ===  

var resolved_ip = dnsResolve(host);


if (isInNet(resolved_ip, "172.31.0.0", "255.255.0.0") ||
    isInNet(resolved_ip, "10.0.1.0", "255.255.255.0"))
    {
    return "proxy1.domain.co.uk:1234"
    }
    if (isInNet(resolved_ip, "192.168.0.0", "255.255.0.0") ||
        isInNet(resolved_ip, "127.0.0.0", "255.0.0.0") ||
        isInNet(resolved_ip, "172.16.0.0", "255.255.0.0") ||
        isInNet(resolved_ip, "172.19.4.0", "255.255.252.0") ||
        ....
        ....
        isInNet(resolved_ip, "172.20.0.0", "255.255.0.0") ||
        isInNet(resolved_ip, "172.21.0.0", "255.255.0.0") ||
        isInNet(resolved_ip, "10.0.0.0", "255.0.0.0"))
    {
    return "DIRECT"

======

It's not a fix, but it's 1001 times better than what we had before, hope this helps someone with a similar situation.
Comment 217 Nickolay_Ponomarev 2011-07-21 00:24:08 PDT
*** Bug 539922 has been marked as a duplicate of this bug. ***
Comment 218 Nickolay_Ponomarev 2011-07-21 00:25:38 PDT
*** Bug 491253 has been marked as a duplicate of this bug. ***
Comment 219 Nickolay_Ponomarev 2011-07-21 00:27:15 PDT
*** Bug 590247 has been marked as a duplicate of this bug. ***
Comment 220 Kosta 2011-07-22 03:07:57 PDT
This bug is not resolved since 2004 year.
People stop chatting here and make it ALL OVER AGAIN FROM SCRATCH.
Apparently the source is a complete mess.
My proposal is to have a one thread tho handle DNS responses, and if there is proxy set then a second thread will be started to resolve the DNS through the proxy.
So the thread must work by sending the DNS request on 52 UDP/TCP ports as it is done is millions of software.
Then the thread waits for response. And the corresponding TAB in the Firefox is waiting for that response, ALL OTHER TABS ARE WORKING JUST FINE - so we can use them as nothing happend.
When the thread(s) receive a response from the DNS server they send a EVENT to the corresponding TAB with the responded information - i.e. the IP of the web server.

That is all.
There is no other matter to discuss.
There is a way to be done in the current Firefox - after all 2004 year is 7 (SEVEN) years old BUG.
Came on get in hands.
Comment 221 Andrew Clements 2011-08-25 16:31:57 PDT
Firefox 7.0 (no addons enabled) - still happening. Am actually forced to use IE because FF unusable behind this proxy. Everything in FF hangs - the About box, the proxy login, the menus etc. It's not frozen - if I wait 20 minutes, a web page will appear. Disable the proxy settings and FF is fine - except no internet of course. Very frustrating & embarrassing.
Comment 222 Jeremy B 2011-09-06 10:15:16 PDT
This is ridiculous. There really shouldn't be any reason why this is still happening. I'm on a clean install of Windows 7 x64 with Firefox 6.0.1 after uninstalling my previous install, deleting every last trace of it, and cleaning my registry. This STILL Happens.

It's really strange. 

I'll be browsing just fine, and then suddenly out of the blue, all of my pages that are loading will get stuck in this loading loop (as if Firefox cannot access the internet.) This is with absolutely no extensions, and just the basic plugins (Flash, Shockwave.)

The only way to get Firefox to work again is to reopen it. While Firefox is hiccuping, I can use Chrome/IE/Opera/Safari just fine.
Comment 223 RNicoletto 2011-09-07 02:38:20 PDT
In reply to comment 220, comment 221 and comment 222: did you try to disable IPv6 via “network.dns.disableIPv6″ user pref?
Comment 224 Jeremy B 2011-09-07 08:47:18 PDT
comen(In reply to RNicoletto from comment #223)
> In reply to comment 220, comment 221 and comment 222: did you try to disable
> IPv6 via “network.dns.disableIPv6″ user pref?

Yep.
Comment 225 Niko L 2011-10-07 00:30:22 PDT
Problem exists with large PAC files


Test 1 - Load a dynamic server page on localhost - A PAC file with about 350 hostname resolves. 

Firefox 7.0.1 with the PAC file enabled (Connection Settings --> Automatic proxy configuration URL): 3.042 s, 2.865 s, 2.892 s, 3.042 s, 3.003 s

Firefox 7.0.1 with no proxy enabled (Connection Settings --> No Proxy): 0.646 s, 0.686 s, 0.588 s, 0.699 s, 0.573 s

Internet Explorer 8 with the PAC file proxy configuration enabled: almost instant meaning < 1 s (not measured)


Firefox GUI with the tested PAC file enabled is completely frozen while loading the page.


Test 2 - Start Firefox with 20 tabs

Firefox 7.0.1 with the PAC file enabled (Connection Settings --> Automatic proxy configuration URL): 2 minutes 30 seconds
Firefox 7.0.1 with no proxy enabled (Connection Settings --> No Proxy): 30 seconds
Comment 226 manujg@gmail.com 2012-01-22 02:02:30 PST
If this is not going to be tracked against a release, then what are the chances of this ever getting fixed?
Comment 227 Benjamin Scherrer 2012-02-02 02:11:06 PST
A resolution for this is crucial especially for enterprise usage where such a proxy configuration is much more common than at home. I urge you to look into this issue for the upcoming enterprise version of firefox (http://blog.mozilla.com/blog/2012/01/10/delivering-a-mozilla-firefox-extended-support-release/).

It would be a huge improvement for us in Firefox, in fact the biggest in years. Thanks!
Comment 228 Mardeg 2012-02-02 02:24:53 PST
(In reply to Benjamin Scherrer from comment #227)
> I urge you to look
> into this issue for the upcoming enterprise version of firefox
> (http://blog.mozilla.com/blog/2012/01/10/delivering-a-mozilla-firefox-
> extended-support-release/).
It's no longer upcoming, it has already been released.
Comment 229 Benjamin Scherrer 2012-02-02 02:28:46 PST
Thanks, I missed the news. So any hopes for progress here? I really hoped it would make this release.
Comment 230 Steve Workman [:sworkman] (INACTIVE) 2012-02-02 09:42:55 PST
All, sorry for the lack of responses on this one.  There have been some other higher priority tasks for various reasons, but this seems to be next on my list of bugs.  I have other tasks as well, though, so I can't devote all my time to it.  I hope to have a patch up sometime in the next month.
Comment 231 Peter 2012-02-03 01:25:12 PST
Well "hopefully next month" is pretty awesome compared to the (almost exactly) 8 years that this bug has been around...
Comment 232 Josh Matthews [:jdm] 2012-02-03 07:34:10 PST
Created attachment 594181 [details] [diff] [review]
"[PAC] Defer proxy resolution for HTTP and HTTPS PAC to avoid blocking main thread during DNS resolution" [a-1.9.1b2=mbeltzner]

I rebased this patch and I'm running it through try to see what the state of the world is.
Comment 233 Josh Matthews [:jdm] 2012-02-03 10:16:29 PST
Created attachment 594233 [details] [diff] [review]
Defer proxy resolution for HTTP and HTTPS PAC to avoid blocking main thread during DNS resolution.
Comment 234 Mozilla RelEng Bot 2012-02-03 10:30:23 PST
Try run for 5fa326c23268 is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=5fa326c23268
Results (out of 99 total builds):
    exception: 4
    success: 39
    warnings: 16
    failure: 40
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/josh@joshmatthews.net-5fa326c23268
Comment 235 Mozilla RelEng Bot 2012-02-03 18:00:25 PST
Try run for 55a64e81467b is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=55a64e81467b
Results (out of 129 total builds):
    success: 114
    warnings: 15
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/josh@joshmatthews.net-55a64e81467b
 Timed out after 06 hours without completing.
Comment 236 Josh Matthews [:jdm] 2012-02-03 18:07:18 PST
That try run was completely green, modulo a couple unrelated known oranges. Jason/Biesi (or Steve, for that matter), does this patch require another round of review? The rebasing I performed was mostly mechanical.
Comment 237 Jason Duell [:jduell] (needinfo me) 2012-02-06 13:34:27 PST
Comment on attachment 594233 [details] [diff] [review]
Defer proxy resolution for HTTP and HTTPS PAC to avoid blocking main thread during DNS resolution.

Steve is going to look the patch over.
Comment 238 Steve Workman [:sworkman] (INACTIVE) 2012-02-06 19:19:26 PST
Jdm, thanks for cleaning up the patch and trying it out.  Looks good to me - no issues.
Comment 240 Adam Pribyl 2012-02-07 00:43:09 PST
Gentleman thank you. It looks like this bug really should be fixed now..? if it gets into next FF release, then it would deserve it's own release party. Is it not the oldest bug ever fixed?
Comment 241 Martin Baranski 2012-02-07 00:51:12 PST
It took Mozilla 8 (verbal: E-I-G-H-T!) years to fix it.

Fixing TNEF attachments in Thunderbird is another sad example for time-consuming bugs and hence is still not fixed.
Maybe fixing proxy resolution will get fixed soon, too?! :)
Comment 242 manujg@gmail.com 2012-02-07 01:29:26 PST
Many thanks for looking at this issue. Much appreciated
Comment 243 Ed Morley [:emorley] 2012-02-07 11:57:47 PST
https://hg.mozilla.org/mozilla-central/rev/20240add1d35
Comment 244 Matthias Versen [:Matti] 2012-02-18 22:05:18 PST
*** Bug 425146 has been marked as a duplicate of this bug. ***
Comment 245 colchaodemola 2012-05-08 05:43:27 PDT
Is this fixed in firefox 12 ? It still hangs if i try to use PAC.
Comment 246 Ryan VanderMeulen [:RyanVM] 2012-05-08 06:10:57 PDT
Per the Target Milestone field above, it is fixed for Firefox 13. You can verify that it is fixed with a current beta release.
Comment 247 Josh Matthews [:jdm] 2012-05-08 09:20:47 PDT
However, as we discovered in bug 750445, this may not actually be fixed in FF 13 either. More investigation is probably required.
Comment 248 Jason Duell [:jduell] (needinfo me) 2012-05-08 12:55:26 PDT
Yes, this is still broken. I filed bug 750445 instead of reopening this one, because the patch for this one stuck (and shouldn't be backed out: we just need to fix it).  Let me know if that's the wrong approach bugzilla-administration-wise.
Comment 249 :Gavin Sharp [email: gavin@gavinsharp.com] 2012-05-08 14:11:34 PDT
That's absolutely the best practice, don't let anyone tell you otherwise :)
Comment 250 Peter da Silva 2012-05-09 05:03:46 PDT
Making everyone who's been following this bug for the past 8 years jump through hoops. Oh well.
Comment 251 :Gavin Sharp [email: gavin@gavinsharp.com] 2012-05-09 10:46:03 PDT
Unfortunately it doesn't really make sense for us to optimize our Bugzilla practices for the casual bug watcher audience (Bugzilla really wasn't designed for that). Trying to track several different changes in a single bug just doesn't work very well.

I suspect that many people on the CC list have since lost interest in getting frequent updates, but those who want to track every step along the way can still CC themselves to the new bug.
Comment 252 Jason Duell [:jduell] (needinfo me) 2012-05-09 14:11:22 PDT
And apologies for this bug not getting fixed--it was an accident in the rebasing of a patch.   For all we knew this was fixed until we discovered otherwise last week.  I'll be looking into it very soon.
Comment 253 Avner 2012-05-09 14:22:28 PDT
No worries, we all moved to Google Chrome anyway. 
But I think I'll stay at the CC list for a while: every time I get an email notification about this defect not being fixed, it reassures me that I made a good decision.
Comment 254 Martin Baranski 2012-05-09 14:40:38 PDT
(In reply to Avner from comment #253)
> No worries, we all moved to Google Chrome anyway. 

I have to confirm this - at least for my colleagues.

Mozilla is way too good (that video about "we believe in freedom" still makes me cry - just beautiful!) to act like this.
Come on guys, step up and get stuff like x84 builds in the release channel done; it's about time!

Remember: its not about the "under the hood", it's about what changes usability and recognizable changes to the user - everything else is just for geeks, not end-users.

And this is definitely true: Firefox is the better browser, because it respects user's privacy like no other. So get that "nerd touch" destroyed by moving faster.
Comment 255 Avner 2012-05-09 14:55:42 PDT
I don't care what goes on under the hood, and I don't care how the proxy server is being resolved technically.
As an end-user, when my browser hangs I just kill it and move to a faster browser.
Comment 256 Martin Baranski 2012-05-09 15:03:04 PDT
That's what I tried to say - thanks for summing it up! :)
Comment 257 Artem Marchenko 2012-05-09 15:06:20 PDT
Indeed. In fact several years ago it was exactly this bug that forced me to move to Chrome. I don't work in the huge company with complex proxy server configs anymore, yet.. I am on Chrome already. Keeping Firefox only for a couple of Chrome-incompatible sites and following this bug out of nostalgia.
Comment 258 Ryan VanderMeulen [:RyanVM] 2012-05-09 15:08:45 PDT
Guys, I know this has been a source of frustration for you, but please show some courtesy to the 103 people CCed to this bug by not spamming them with your reasons for switching to Chrome.
Comment 259 Martin Baranski 2012-05-09 15:34:46 PDT
This is my last comment, because I want to respect Ryans kind wish to show some courtesy:

I'm not on Chrome, I'm on Firefox (even Nightly x64 right now) - since Version 1.0, at work, too, even on Android and until it really brakes the basic rules, I guess.

So, finally: sorry to those 103 people for spamming. I did not intend to do so - I just wanted to kind of "resurrect" the spirit that once brought so many users from IE to the great and lovely fox.
Comment 260 Peter da Silva 2012-05-10 04:41:55 PDT
I stick with Firefox, and I really appreciate how it has kept listening to its users and stuck with a clean, native user interface... but there's a couple of old horrible bugs like this that really need to get fixed, and I keep following them in the (apparently not quite vain) hope that it will eventually happen. The biggest one is the incomplete downloads bug, where the remote end closes the connection before the download is finished and there's no indication on this end that you only got 20MB of an expected 40MB. Is that another "forgotten" one? If the download manager would just leave the expected size visible after the download ends we'd at least have a chance of catching it when it happens.
Comment 261 Patrick McManus [:mcmanus] 2012-07-24 19:48:33 PDT
backed out in bug 767005
Comment 262 Laszlo Ersek 2012-08-05 13:32:40 PDT
Since the fix has been backed out, please consider reopening this BZ. Thanks!
Comment 263 Patrick McManus [:mcmanus] 2012-08-05 13:50:04 PDT
(In reply to Laszlo Ersek from comment #262)
> Since the fix has been backed out, please consider reopening this BZ. Thanks!

it would just be a dup of the work being done in bug 769764, you can follow it over there.
Comment 264 Marco Castelluccio [:marco] (PTO until August 24/25) 2012-09-14 14:03:44 PDT

*** This bug has been marked as a duplicate of bug 769764 ***

Note You need to log in before you can comment on or make changes to this bug.