Closed
Bug 84264
Opened 23 years ago
Closed 23 years ago
need to reuse connections to different hosts when using a proxy server (keep-alive)
Categories
(Core :: Networking: HTTP, defect)
Tracking
()
VERIFIED
FIXED
mozilla0.9.4
People
(Reporter: bob+mozilla, Assigned: bbaetz)
References
()
Details
(Keywords: topembed, Whiteboard: [topembed+])
Attachments
(1 file)
1.55 KB,
patch
|
Details | Diff | Splinter Review |
With a HTTP/1.1 proxy, mozilla closes all open connections after loading each page. i.e. it treats keep-alive connections to the proxy as if it were talking to the server. I have a proxy that tries to keep-alive as many connections to the browser as possible to improve speed, but mozilla closes them all when it's done with a particular page, despite the proxy sending keep-alive headers. It appears that mozilla keeps the connection open until the next page is loaded, at which time it closes all open connections and opens a bunch of new ones for the new page. In case that's not clear...here is a log snippet from my proxy: [13468 Tue Jun 5 23:24:45 2001] New proxy connection accepted from chani:35192 [13468 Tue Jun 5 23:24:45 2001] HTTP/1.1 proxy request for http://www.kuro5hin.org/ received. [13468 Tue Jun 5 23:24:53 2001] [200 OK] for http://www.kuro5hin.org/ (text/html 118256 bytes) [13497 Tue Jun 5 23:25:11 2001] New proxy connection accepted from chani:35194 [13497 Tue Jun 5 23:25:11 2001] HTTP/1.1 proxy request for http://slashdot.org/index.pl received. [13468 Tue Jun 5 23:25:11 2001] Exiting for client chani:35192 (1 requests processed) {Client closed} [13497 Tue Jun 5 23:25:13 2001] [200 OK] for http://slashdot.org/index.pl (text/html unspecified bytes) Notice the following: 1) Mozilla opens a new connection to the proxy to request a new page, rather than using an existing open keep-alive connection to the proxy. 2) Mozilla closes the first connection *after* it receives the request for the next page. (the leftmost number in the log is the pid of the server handling the request -- this is a forking proxy server) The above obeservations also hold when the maximum number of connections to the proxy are open (8?) -- for pages with lots of images. In that case *all* existing connections except the new one are closed immediately after the new request is received. Verified with 6/4/2001 nightly. Here are the headers sent by the proxy to mozilla with the response (from slashdot.org): Cache-Control: private Connection: keep-alive Date: Wed, 06 Jun 2001 04:29:16 GMT Pragma: no-cache Transfer-Encoding: chunked Via: 1.1 chani.access.services.wisc.edu Server: Apache/1.3.12 (Unix) mod_perl/1.24 Content-Encoding: gzip Content-Type: text/html Client-Date: Wed, 06 Jun 2001 04:28:29 GMT Client-Peer: 64.28.67.150:80 Keep-Alive: 300
Summary: mozilla closes connections prematurely with HTTP/1.1 proxy using keep-alive → mozilla closes connections prematurely with HTTP/1.1 proxy using keep-alive
Reporter | ||
Comment 4•23 years ago
|
||
Behavior is same with nightly from 6/22 (build ID 2001062212). I'll try a newer nightly when I have time next week.
after looking at this bug again it sounds like bug 87047. Reporter could you look at that bug and see if this is a duplicate of, or depends on that bug?
Reporter | ||
Comment 6•23 years ago
|
||
Undoubtedly the issues in bug 87047 lie in the same piece of code as this one. However, the behavior is different. In this case, Preferences->Debug->Networking->Enable Keep Alive is ON (bug 87047 complains that mozilla sends Connection: close when it is OFF). For this case, mozilla correctly sends the Connection: keep-alive header, but it forcibly closes the connection anyway. It even keeps the connection open for a small time. The connection is forcibly closed when the *next* request is received. I'm not an expert on bugzilla, but if the above description sounds like a "dependency" for bugzilla's purposes, go ahead and change things to reflect that.
Comment 7•23 years ago
|
||
Marking NEW.
Removing depend (ksosez: please read bugs carefully before adding depends) Even though this bug doesn't depend on bug 87047 it affects the same code.
No longer depends on: 87047
Comment 10•23 years ago
|
||
Bob: please download and test again after bug 87047 is fixed
Depends on: 87047
Reporter | ||
Comment 11•23 years ago
|
||
Testing with nightly build ID 2001071709. Behavior is unchanged. I'm not sure if this build includes the fix for bug 87047. I'll try again after tomorrow's builds come out.
Comment 12•23 years ago
|
||
*** Bug 91474 has been marked as a duplicate of this bug. ***
Reporter | ||
Comment 13•23 years ago
|
||
Behavior is still unchanged with 2001071906 nightly. But I see that bug 87047 has been reopened too...
Assignee | ||
Comment 14•23 years ago
|
||
Really? Are you sure? I was just about to mark this as closed. bug 87407 was reopened for possible checkin on the branch - the fix is in on the trunk. What proxy server are you using? I verified that my fix worked with an http/1.1 proxy server, and with various http/1.1 web servers (apache, iis, www.google.com, netscape's web server and a few top100 pages) Taking bug. We currently consider a 1.1 connection to be timed out after 10 seconds of inactivity. Maybe that number should be increased (see bug 91204)
Assignee: neeti → bbaetz
Reporter | ||
Comment 16•23 years ago
|
||
> We currently consider a 1.1 connection to be timed out after 10 seconds of > inactivity. Maybe that number should be increased (see bug 91204) My proxy (http://draal.physics.wisc.edu/FilterProxy/) implements the 'Connection: keep-alive' and 'Keep-Alive: <seconds>' headers as specified in rfc2068. According to that rfc, the argument of 'Connection' specifies other headers which describe the connection for this hop. The 'Keep-Alive' header specifies how long the client (or server) wants to keep the connection open. Mozilla is sending 'Keep-Alive: 300' with every outgoing request, and therefore should be keeping the connection open for 300 seconds, not 10! rfc2616 does not mention the Keep-Alive header (RFC 2616 obsoletes 2068), so it would be OK to close the connection after 10 seconds *if* Mozilla did not send the header 'Keep-Alive: 300', nor 'Connection: keep-alive', then it would be conforming to rfc2616. Looking at the headers sent by nightly 2001071906, I see that it is now sending 'Proxy-Connection: keep-alive', *no* 'Connection' header, and 'Keep-Alive: 300'! Correct me if I'm wrong, but the 'Proxy-Connection' header is a netscape 4.x hack on top of HTTP/1.0 to make persistent HTTP/1.0 proxy connections, and isn't described by *any* RFC! (I checked - it does not appear in rfc2616) Why is Mozilla sending this header? My proxy obeys this header only if the client is speaking HTTP/1.0 and identifies itself as Netscape 4.x, but for HTTP/1.1 it looks for the 'Connection' header if present (it's optional, right?). HTTP/1.1 connections are by default persistent, so Proxy-Connection is meaningless (Connection and Keep-Alive headers are per-hop anyway, so they are meant for the proxy if a proxy is configured). Here are the headers Mozilla is sending: GET http://www.advogato.org/ HTTP/1.1 Host: www.advogato.org User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.2+) Gecko/20010719 Accept: text/xml, application/xml, application/xhtml+xml, text/html;q=0.9, image/png, image/jpeg, image/gif;q=0.2, text/plain;q=0.8, text/css, */*;q=0.1 Accept-Language: en-us Accept-Encoding: gzip,deflate,compress,identity Accept-Charset: ISO-8859-1, utf-8;q=0.66, *;q=0.66 Keep-Alive: 300 Proxy-Connection: keep-alive Proxy-Authorization: Basic <removed> My local rfc mirror seems to be down, and I don't have a local copy of rfc2068. The above is from memory, please correct me if I'm wrong. One of our local gateways seems to be acting up, and I can't connect to much of anywhere. :( I'll more carefully test how Mozilla is handling connections when I get home tonight... My goal here is to keep as many connections open as possible, for as long as possible between Mozilla and the proxy, to cut down on network load, and proxy-server load (fork-per-connection proxy...).
Assignee | ||
Comment 17•23 years ago
|
||
(Liberal snipping applied) > Mozilla is sending 'Keep-Alive: 300' with every outgoing request, and > therefore should be keeping the connection open for 300 seconds, not 10! Oh, drat. Thanks for that. Let me generate a patch, and check this in as soon as the tree opens (and change that number to something more sensible, like 25, while I'm at it). We do need to redo this entire timeout stuff, though - see bug 91024. I'll attach the patch to bug 87047, and cc you on that. Actually, what happens is that we (as of my checkin on monday) try to be conformant to both specs. We send those headers for compatability with 1.0 servers, but if we receive a 1.1 response, without those headers we will use keep-alive. If a server (1.0 or 1.1) sends a timeout in its keep-alive header, then we will use that timeout. Our default is only used it no timeout is specified (as for 1.1 persistent connections) Re proxy-connection: The reason we do this is set out in bug 87047, and rfc2616 section 19.6.2. There are problems with transparent proxies, because of this. We're sending what IE does, though. I'll download your proxy server and try it out, and see what I can work out.
Target Milestone: mozilla1.0 → mozilla0.9.3
Reporter | ||
Comment 18•23 years ago
|
||
Can the Keep-Alive timeout be made a configurable option? Maybe put it in Preferences->Advanced->Proxies? Or an option in prefs.js would be fine too. Short timeouts make sense for servers, and long ones for proxies. > If a server (1.0 or 1.1) sends a timeout in its keep-alive header, then we > will use that timeout. Our default is only used it no timeout is specified (as > for 1.1 persistent connections) That would be great! (but that's not what's happening -- and why I filed this bug) Then the proxy could tell it to keep-alive as long as it wants and a mozilla preference isn't necessary (for me). Proxy-Connection headers shouldn't be a problem since all HTTP/1.1 servers/proxies should ignore them. (I think IE sending Proxy-Connection and no Connection headers is Just Plain Wrong) But will sending both Connection and Proxy-Connection confuse HTTP/1.0 servers? Note that this bug is about Mozilla forcibly (no 'Connection: close' header) closing all existing connections when a new page is loaded. i.e. if I load http://slashdot.org, the page and all its associated images are loaded properly, using persistent connections. If I then hit one of my bookmarks, as soon as Mozilla sees the new GET request (and it uses a NEW connection for the new request), it closes all existing connections and loads the new page. AFAICT, Keep-Alive stuff works fine for a single page and its associated images.
Assignee | ||
Comment 19•23 years ago
|
||
It is a pref - network.http.keep-alive.timeout. 300 is what we've been using all
along, so I figure it can't break anything. As I said, we're going to revisit
this next week, so please comment in that other bug.
> Note that this bug is about Mozilla forcibly (no 'Connection: close' header)
> closing all existing connections when a new page is loaded.
Ah, I missed that ('new page') the first time. That was my guess, actually
(we're considering the two servers as different, and not taking proxies into
account) - let me play with it a bit this afternoon, and see.
Assignee | ||
Comment 20•23 years ago
|
||
updating summary to better cover what this bug has morphed to. I'm working on a patch now
Summary: mozilla closes connections prematurely with HTTP/1.1 proxy using keep-alive → need to reuse connections to different hosts when using a proxy server
Comment 21•23 years ago
|
||
Old summary: "need to reuse connections to different hosts when using a proxy server" New summary: "need to reuse connections to different hosts when using a proxy server (keep-alive)" Changed only to make dupe-finding easier.
Summary: need to reuse connections to different hosts when using a proxy server → need to reuse connections to different hosts when using a proxy server (keep-alive)
Assignee | ||
Comment 22•23 years ago
|
||
Comment 23•23 years ago
|
||
r=gagan
Assignee | ||
Comment 24•23 years ago
|
||
I tried testing that patch on the specific proxy server mentioned here, but any use of the proxy server seemed to start an endless fork bomb, and I had to reboot my machine. The docs did warn that it may not work properly with perl 5.6.... It should fix the problem (I tested with apache, and squid, and a socks proxy as well), but I'd appreciate confirmation.
Assignee | ||
Comment 25•23 years ago
|
||
dougt says sr if I remove the else
Assignee | ||
Comment 26•23 years ago
|
||
Checked in on trunk.
Reporter | ||
Comment 27•23 years ago
|
||
Your patch works great! Tested with CVS from this morning, mozilla never closes connections to my proxy! Thanks for your work Bradley! I tested the patch as attached, dunno about removing the else, dunno about "sr". Must be secret netscape-talk. ;)
Comment 28•23 years ago
|
||
heh, no secret codes here. Just short for "super reviewed". And it's mozilla, not netscape.
Reporter | ||
Comment 29•23 years ago
|
||
The 7/20 nightly (build ID 2001072008) has worse problems (I think after Bradley's patch was checked in?). This build doesn't close connections to the proxy any longer, but it also doesn't re-use the connections that exist. Every time I hit a new entry in my bookmarks, Mozilla opens a new connection to the proxy. This results in the number of connections growing linearly as you browse. Perhaps this is the source of the "fork-bomb" you saw Bradley? The patch I tested worked great, did re-use connections, and only ever opens 8 connections to the proxy, continuing to re-use them. (Is the difference between the 7/20 nightly and your patch the "else" statement? Could that be causing this?) Also I'm not sure Mozilla is obeying the time in the Keep-Alive header. It keeps the connection open for some time (about 8 minutes), while the Keep-Alive time sent to it by the proxy is 5 minutes. Not a big deal, but might be indicative of another problem...
Assignee | ||
Comment 30•23 years ago
|
||
No, dougt just wanted to me to change it from: if (foo) { return ... } else { return ... } to if (foo) { return ... } return ... style nit only; no change in what happens. However, the 0720 builds don't have my patch in it - I only checked it in that afternoon. If you see this with later builds, then I'll need to look into it. Also, we attempt to close connections only when opening other ones, so if you left the browser open without going to any other http sites, we would keep it open beyond 5 minutes of idle time.
Reporter | ||
Comment 31•23 years ago
|
||
Build 2001072206 works great. Sorry for any confusion. Closing connections on browser activity makes sense. All the (close time - last request time) times I've seen have been greater than the Keep-Alive time, and that's peachy.
Comment 32•23 years ago
|
||
waiting for browser activity to close does *not* make sense. long timeouts (and these could be tens of minutes or hours!) force servers to do the active close. Active close means CLOSE_WAIT and that socket is wasted for 2MSL. Sockets are much more precious on the server side than they are on the client side. Very busy servers do run out of sockets. Please be kind. Only the client can know or make the best guess as to when that socket will be reused. Reward servers who trust you to make that decision (instead of just applying 3 second timeouts which is common) - don't punish them by wasting a socket entry for minutes and then not even reusing it. The end result of this behavior will be servers that set very low timeouts for mozilla clients ruining persistence all together. This used to be very common before MSIEs fairly short timeouts beacuse defacto standard. MSIE btw - still gets many more connection reuses than mozilla. Persistence is great, really long idled persistence is generally not worth the cost paid in statefulness to the server. If the server offers a gigantic keep-alive value, that is a totally different story. Servers (proxies or origin) who wish this should pursue that strategy and mozilla could honor that.. in low demand situations the statefull cost is no big deal.
Assignee | ||
Comment 33•23 years ago
|
||
No, you misunderstand. A connection to _any_ other server will expire entries past the keep-alive timeout. (we can't know whether the user will return to that site 1 second before the timeout). Further comments should be made in bug 91204.
Assignee | ||
Comment 34•23 years ago
|
||
*** Bug 90509 has been marked as a duplicate of this bug. ***
Assignee | ||
Comment 35•23 years ago
|
||
pushing out milestone to avoid getting spammed - this is checked in on the trunk already, and so the milestone doesn't really make sense.
Target Milestone: mozilla0.9.3 → mozilla0.9.4
Reporter | ||
Comment 36•23 years ago
|
||
I'm beginning to think the current behavior is a little nonsensical. Mozilla waits for user activity before closing timed-out connections to the proxy. In other words, the user waits for a while, and loads a new page. Mozilla says "hey, I need to make a new HTTP request. Let's close all the connections to the proxy first." ...which doesn't really make sense. If there's a connection open, wny not use it? Closing them all first and reopening them just puts a lot of load on the proxy. Secondly, I'm seeing behavior like this: 1) Mozilla times out, closes all connections. 2) User attempts to load new page 3) Mozilla DOESN'T SEND A HTTP REQUEST. I've watched with ngrep, and in fact Mozilla is not sending any data to the proxy (it has no connections open to the proxy either). This happens after loading ~5-10 pages, though I cannot trigger it reliably. Is there a bug for this? (It existed before the fix for 84264 went in, but is obviously related).
Assignee | ||
Comment 37•23 years ago
|
||
the first bit is known, but I don't know if theres a bug on it. The second bit is strange. If you can trigger it with http logging enabled (you'd need a debug build), or reliably reproduce it, then please file a new bug.
Reporter | ||
Comment 38•23 years ago
|
||
The first bit (re-using timed out, but open connections) seems to go with this bug. For the second bit, I can reproduce it by: 1) load a page (I use http://www.mozillazine.org for this test) 2) wait for it to complete loading 3) hit reload. When using a http/1.1 proxy, the page itself loads fine, then mozilla sends requests for (most of) the images. On the url above, the proxy returns 304 (not modified) for all images. There are a handful of images on the page though that mozilla does *not* send new requests for. Each of these is *not* redrawn, and mozilla hangs (the throbber never stops spinning). The set of images it "forgets" to request is the same each time (tile.jpg, grass.gif, a few others...) I then hit 'esc' or the stop button to stop the throbber. Subsequently, mozilla won't load any pages. Hitting bookmarks or typing something into the url bar doesn't generate new requests to the proxy. (I'm watching the network traffic with ngrep -- this can't be a proxy bug) This doesn't happen if the proxy is not enabled. I'm going to file a new bug for this second one, unless someone tells me otherwise.
Assignee | ||
Comment 39•23 years ago
|
||
Checked in on 092 branch.
Status: ASSIGNED → RESOLVED
Closed: 23 years ago
Keywords: vbranch
Resolution: --- → FIXED
Whiteboard: checked in on trunk → [topembed+]
You need to log in
before you can comment on or make changes to this bug.
Description
•