need to reuse connections to different hosts when using a proxy server (keep-alive)

VERIFIED FIXED in mozilla0.9.4

Status

()

Core
Networking: HTTP
VERIFIED FIXED
17 years ago
4 years ago

People

(Reporter: Bob McElrath, Assigned: bbaetz)

Tracking

({topembed})

Trunk
mozilla0.9.4
x86
Linux
topembed
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [topembed+], URL)

Attachments

(1 attachment)

(Reporter)

Description

17 years ago
With a HTTP/1.1 proxy, mozilla closes all open connections after loading each
page.  i.e. it treats keep-alive connections to the proxy as if it were talking
to the server.  I have a proxy that tries to keep-alive as many connections to
the browser as possible to improve speed, but mozilla closes them all when it's
done with a particular page, despite the proxy sending keep-alive headers.  It
appears that mozilla keeps the connection open until the next page is loaded, at
which time it closes all open connections and opens a bunch of new ones for the
new page.  In case that's not clear...here is a log snippet from my proxy:


[13468 Tue Jun  5 23:24:45 2001] New proxy connection accepted from chani:35192
[13468 Tue Jun  5 23:24:45 2001] HTTP/1.1 proxy request for
http://www.kuro5hin.org/ received.
[13468 Tue Jun  5 23:24:53 2001] [200 OK] for http://www.kuro5hin.org/
(text/html 118256 bytes)
[13497 Tue Jun  5 23:25:11 2001] New proxy connection accepted from chani:35194
[13497 Tue Jun  5 23:25:11 2001] HTTP/1.1 proxy request for
http://slashdot.org/index.pl received.
[13468 Tue Jun  5 23:25:11 2001] Exiting for client chani:35192 (1 requests
processed) {Client closed}
[13497 Tue Jun  5 23:25:13 2001] [200 OK] for http://slashdot.org/index.pl
(text/html unspecified bytes)


Notice the following:
1) Mozilla opens a new connection to the proxy to request a new page, rather
than using an existing open keep-alive connection to the proxy.
2) Mozilla closes the first connection *after* it receives the request for the
next page.  (the leftmost number in the log is the pid of the server handling
the request -- this is a forking proxy server)

The above obeservations also hold when the maximum number of connections to the
proxy are open (8?) -- for pages with lots of images.  In that case *all*
existing connections except the new one are closed immediately after the new
request is received.  Verified with 6/4/2001 nightly.  Here are the headers sent
by the proxy to mozilla with the response (from slashdot.org):
Cache-Control: private
Connection: keep-alive
Date: Wed, 06 Jun 2001 04:29:16 GMT
Pragma: no-cache
Transfer-Encoding: chunked
Via: 1.1 chani.access.services.wisc.edu
Server: Apache/1.3.12 (Unix) mod_perl/1.24
Content-Encoding: gzip
Content-Type: text/html
Client-Date: Wed, 06 Jun 2001 04:28:29 GMT
Client-Peer: 64.28.67.150:80
Keep-Alive: 300

Updated

17 years ago
Summary: mozilla closes connections prematurely with HTTP/1.1 proxy using keep-alive → mozilla closes connections prematurely with HTTP/1.1 proxy using keep-alive
May be related to bug 82873. The new HTTP is still work in progress.

Comment 2

17 years ago
See also bug 78724.

Comment 3

17 years ago
reporter: does this still happen in a later build?
(Reporter)

Comment 4

17 years ago
Behavior is same with nightly from 6/22 (build ID 2001062212).  I'll try a newer
nightly when I have time next week.

Comment 5

17 years ago
after looking at this bug again it sounds like bug 87047. Reporter could you
look at that bug and see if this is a duplicate of, or depends on that bug?
(Reporter)

Comment 6

17 years ago
Undoubtedly the issues in bug 87047 lie in the same piece of code as this one. 
However, the behavior is different.  In this case,
Preferences->Debug->Networking->Enable Keep Alive is ON (bug 87047 complains
that mozilla sends Connection: close when it is OFF).  For this case, mozilla
correctly sends the Connection: keep-alive header, but it forcibly closes the
connection anyway.  It even keeps the connection open for a small time.  The
connection is forcibly closed when the *next* request is received.

I'm not an expert on bugzilla, but if the above description sounds like a
"dependency" for bugzilla's purposes, go ahead and change things to reflect that.

Comment 7

17 years ago
Marking NEW.
Status: UNCONFIRMED → NEW
Depends on: 87047
Ever confirmed: true

Comment 8

17 years ago
adjusting milestone.
Target Milestone: --- → mozilla1.0

Comment 9

17 years ago
Removing depend (ksosez: please read bugs carefully before adding depends)

Even though this bug doesn't depend on bug 87047 it affects the same code.
No longer depends on: 87047

Comment 10

17 years ago
Bob: please download and test again after bug 87047 is fixed
Depends on: 87047
(Reporter)

Comment 11

17 years ago
Testing with nightly build ID 2001071709.  Behavior is unchanged.  I'm not sure
if this build includes the fix for bug 87047.  I'll try again after tomorrow's
builds come out.
*** Bug 91474 has been marked as a duplicate of this bug. ***
(Reporter)

Comment 13

17 years ago
Behavior is still unchanged with 2001071906 nightly.  But I see that bug 87047
has been reopened too...
(Assignee)

Comment 14

17 years ago
Really? Are you sure? I was just about to mark this as closed.

bug 87407 was reopened for possible checkin on the branch - the fix is in on the
trunk.

What proxy server are you using? I verified that my fix worked with an http/1.1
proxy server, and with various http/1.1 web servers (apache, iis,
www.google.com, netscape's web server and a few top100 pages)

Taking bug.

We currently consider a 1.1 connection to be timed out after 10 seconds of
inactivity. Maybe that number should be increased (see bug 91204)
Assignee: neeti → bbaetz
(Assignee)

Comment 15

17 years ago
That should be bug 87047, sorry.
Status: NEW → ASSIGNED
(Reporter)

Comment 16

17 years ago
> We currently consider a 1.1 connection to be timed out after 10 seconds of
> inactivity. Maybe that number should be increased (see bug 91204)

My proxy (http://draal.physics.wisc.edu/FilterProxy/) implements the
'Connection: keep-alive' and 'Keep-Alive: <seconds>' headers as specified in
rfc2068.  According to that rfc, the argument of 'Connection' specifies other
headers which describe the connection for this hop.  The 'Keep-Alive' header
specifies how long the client (or server) wants to keep the connection open.
Mozilla is sending 'Keep-Alive: 300' with every outgoing request, and therefore
should be keeping the connection open for 300 seconds, not 10!  rfc2616 does not
mention the Keep-Alive header (RFC 2616 obsoletes 2068), so it would be OK to
close the connection after 10 seconds *if* Mozilla did not send the header
'Keep-Alive: 300', nor 'Connection: keep-alive', then it would be conforming to
rfc2616.

Looking at the headers sent by nightly 2001071906, I see that it is now sending
'Proxy-Connection: keep-alive', *no* 'Connection' header, and 'Keep-Alive:
300'!  Correct me if I'm wrong, but the 'Proxy-Connection' header is a netscape
4.x hack on top of HTTP/1.0 to make persistent HTTP/1.0 proxy connections, and
isn't described by *any* RFC!  (I checked - it does not appear in rfc2616) Why
is Mozilla sending this header?  My proxy obeys this header only if the client
is speaking HTTP/1.0 and identifies itself as Netscape 4.x, but for HTTP/1.1 it
looks for the 'Connection' header if present (it's optional, right?).  HTTP/1.1
connections are by default persistent, so Proxy-Connection is meaningless
(Connection and Keep-Alive headers are per-hop anyway, so they are meant for the
proxy if a proxy is configured).  Here are the headers Mozilla is sending:

GET http://www.advogato.org/ HTTP/1.1
Host: www.advogato.org
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.2+) Gecko/20010719
Accept: text/xml, application/xml, application/xhtml+xml, text/html;q=0.9,
image/png, image/jpeg, image/gif;q=0.2, text/plain;q=0.8, text/css, */*;q=0.1
Accept-Language: en-us
Accept-Encoding: gzip,deflate,compress,identity
Accept-Charset: ISO-8859-1, utf-8;q=0.66, *;q=0.66
Keep-Alive: 300
Proxy-Connection: keep-alive
Proxy-Authorization: Basic <removed>

My local rfc mirror seems to be down, and I don't have a local copy of rfc2068.
The above is from memory, please correct me if I'm wrong.  One of our local
gateways seems to be acting up, and I can't connect to much of anywhere.  :(
I'll more carefully test how Mozilla is handling connections when I get home
tonight...

My goal here is to keep as many connections open as possible, for as long as
possible between Mozilla and the proxy, to cut down on network load, and
proxy-server load (fork-per-connection proxy...).
(Assignee)

Comment 17

17 years ago
(Liberal snipping applied)

> Mozilla is sending 'Keep-Alive: 300' with every outgoing request, and
> therefore should be keeping the connection open for 300 seconds, not 10! 

Oh, drat. Thanks for that. Let me generate a patch, and check this in as soon as
the tree opens (and change that number to something more sensible, like 25,
while I'm at it). We do need to redo this entire timeout stuff, though - see bug
91024. I'll attach the patch to bug 87047, and cc you on that.

Actually, what happens is that we (as of my checkin on monday) try to be
conformant to both specs. We send those headers for compatability with 1.0
servers, but if we receive a 1.1 response, without those headers we will use
keep-alive.

If a server (1.0 or 1.1) sends a timeout in its keep-alive header, then we will
use that timeout. Our default is only used it no timeout is specified (as for
1.1 persistent connections)

Re proxy-connection: The reason we do this is set out in bug 87047, and rfc2616
section 19.6.2. There are problems with transparent proxies, because of this.
We're sending what IE does, though.

I'll download your proxy server and try it out, and see what I can work out.
Target Milestone: mozilla1.0 → mozilla0.9.3
(Reporter)

Comment 18

17 years ago
Can the Keep-Alive timeout be made a configurable option?  Maybe put it in
Preferences->Advanced->Proxies?  Or an option in prefs.js would be fine too. 
Short timeouts make sense for servers, and long ones for proxies.

> If a server (1.0 or 1.1) sends a timeout in its keep-alive header, then we 
> will use that timeout. Our default is only used it no timeout is specified (as
> for 1.1 persistent connections)

That would be great!  (but that's not what's happening -- and why I filed this
bug)  Then the proxy could tell it to keep-alive as long as it wants and a
mozilla preference isn't necessary (for me).

Proxy-Connection headers shouldn't be a problem since all HTTP/1.1
servers/proxies  should ignore them.  (I think IE sending Proxy-Connection and
no Connection headers is Just Plain Wrong)  But will sending both Connection and
Proxy-Connection confuse HTTP/1.0 servers?

Note that this bug is about Mozilla forcibly (no 'Connection: close' header)
closing all existing connections when a new page is loaded.  i.e. if I load
http://slashdot.org, the page and all its associated images are loaded properly,
using persistent connections.  If I then hit one of my bookmarks, as soon as
Mozilla sees the new GET request (and it uses a NEW connection for the new
request), it closes all existing connections and loads the new page.  AFAICT,
Keep-Alive stuff works fine for a single page and its associated images.
(Assignee)

Comment 19

17 years ago
It is a pref - network.http.keep-alive.timeout. 300 is what we've been using all
along, so I figure it can't break anything. As I said, we're going to revisit
this next week, so please comment in that other bug.

> Note that this bug is about Mozilla forcibly (no 'Connection: close' header)
> closing all existing connections when a new page is loaded. 

Ah, I missed that ('new page') the first time. That was my guess, actually
(we're considering the two servers as different, and not taking proxies into
account) - let me play with it a bit this afternoon, and see.
(Assignee)

Comment 20

17 years ago
updating summary to better cover what this bug has morphed to. I'm working on a
patch now
Summary: mozilla closes connections prematurely with HTTP/1.1 proxy using keep-alive → need to reuse connections to different hosts when using a proxy server
Old summary:
"need to reuse connections to different hosts when using a proxy server"

New summary:
"need to reuse connections to different hosts when using a proxy server
(keep-alive)"

Changed only to make dupe-finding easier.
Summary: need to reuse connections to different hosts when using a proxy server → need to reuse connections to different hosts when using a proxy server (keep-alive)
(Assignee)

Comment 22

17 years ago
Created attachment 42928 [details] [diff] [review]
patch

Comment 23

17 years ago
r=gagan
(Assignee)

Comment 24

17 years ago
I tried testing that patch on the specific proxy server mentioned here, but any
use of the proxy server seemed to start an endless fork bomb, and I had to
reboot my machine.

The docs did warn that it may not work properly with perl 5.6.... It should fix
the problem (I tested with apache, and squid, and a socks proxy as well), but
I'd appreciate confirmation.
(Assignee)

Comment 25

17 years ago
dougt says sr if I remove the else

Updated

17 years ago
Keywords: topembed
(Assignee)

Comment 26

17 years ago
Checked in on trunk.
Keywords: nsBranch, vtrunk
Whiteboard: checked in on trunk
(Reporter)

Comment 27

17 years ago
Your patch works great!  Tested with CVS from this morning, mozilla never closes
connections to my proxy!  Thanks for your work Bradley!

I tested the patch as attached, dunno about removing the else, dunno about "sr".
 Must be secret netscape-talk.  ;)

Comment 28

17 years ago
heh, no secret codes here. Just short for "super reviewed". And it's mozilla,
not netscape.
(Reporter)

Comment 29

17 years ago
The 7/20 nightly (build ID 2001072008) has worse problems (I think after
Bradley's patch was checked in?).  This build doesn't close connections to the
proxy any longer, but it also doesn't re-use the connections that exist.  Every
time I hit a  new entry in my bookmarks, Mozilla opens a new connection to the
proxy.  This results in the number of connections growing linearly as you
browse.  Perhaps this is the source of the "fork-bomb" you saw Bradley?

The patch I tested worked great, did re-use connections, and only ever opens 8
connections to the proxy, continuing to re-use them.  (Is the difference between
the 7/20 nightly and your patch the "else" statement?  Could that be causing this?)

Also I'm not sure Mozilla is obeying the time in the Keep-Alive header.  It
keeps the connection open for some time (about 8 minutes), while the Keep-Alive
time sent to it by the proxy is 5 minutes.  Not a big deal, but might be
indicative of another problem...
(Assignee)

Comment 30

17 years ago
No, dougt just wanted to me to change it from:

if (foo) {
return ...
} else {
return ...
}
to

if (foo) {
return ...
}
return ...

style nit only; no change in what happens.

However, the 0720 builds don't have my patch in it - I only checked it in that
afternoon. If you see this with later builds, then I'll need to look into it.

Also, we attempt to close connections only when opening other ones, so if you
left the browser open without going to any other http sites, we would keep it
open beyond 5 minutes of idle time.
(Reporter)

Comment 31

17 years ago
Build 2001072206 works great.  Sorry for any confusion.

Closing connections on browser activity makes sense.  All the (close time - last
request time) times I've seen have been greater than the Keep-Alive time, and
that's peachy.
waiting for browser activity to close does *not* make sense. long timeouts (and
these could be tens of minutes or hours!) force servers to do the active close.
Active close means CLOSE_WAIT and that socket is wasted for 2MSL. Sockets 
are much more precious on the server side than they are on the client side.
Very busy servers do run out of sockets.

Please be kind. Only the client can know or make the best guess as to when
that socket will be reused. Reward servers who trust you to make that decision
(instead of just applying 3 second timeouts which is common) - don't punish them
by wasting a socket entry for minutes and then not even reusing it.

The end result of this behavior will be servers that set very low timeouts
for mozilla clients ruining persistence all together. This used to be
very common before MSIEs fairly short timeouts beacuse defacto standard.
MSIE btw - still gets many more connection reuses than mozilla.

Persistence is great, really long idled persistence is generally not worth the
cost paid in statefulness to the server. 

If the server offers a gigantic keep-alive value, that is a totally different
story. Servers (proxies or origin) who wish this should pursue that strategy
and mozilla could honor that.. in low demand situations the statefull cost
is no big deal. 
(Assignee)

Comment 33

17 years ago
No, you misunderstand. A connection to _any_ other server will expire entries
past the keep-alive timeout. (we can't know whether the user will return to that
site 1 second before the timeout). Further comments should be made in bug 91204.
(Assignee)

Comment 34

17 years ago
*** Bug 90509 has been marked as a duplicate of this bug. ***
(Assignee)

Updated

17 years ago
Depends on: 86973
(Assignee)

Updated

17 years ago
No longer depends on: 86973
(Assignee)

Comment 35

17 years ago
pushing out milestone to avoid getting spammed - this is checked in on the trunk
already, and so the milestone doesn't really make sense.
Target Milestone: mozilla0.9.3 → mozilla0.9.4
(Reporter)

Comment 36

17 years ago
I'm beginning to think the current behavior is a little nonsensical.  Mozilla
waits for user activity before closing timed-out connections to the proxy.  In
other words, the user waits for a while, and loads a new page.  Mozilla says
"hey, I need to make a new HTTP request.  Let's close all the connections to the
proxy first."  ...which doesn't really make sense.  If there's a connection
open, wny not use it?  Closing them all first and reopening them just puts a lot
of load on the proxy.

Secondly, I'm seeing behavior like this:
  1) Mozilla times out, closes all connections.
  2) User attempts to load new page
  3) Mozilla DOESN'T SEND A HTTP REQUEST.
I've watched with ngrep, and in fact Mozilla is not sending any data to the
proxy (it has no connections open to the proxy either).  This happens after
loading ~5-10 pages, though I cannot trigger it reliably.

Is there a bug for this?  (It existed before the fix for 84264 went in, but is
obviously related).
(Assignee)

Comment 37

17 years ago
the first bit is known, but I don't know if theres a bug on it.

The second bit is strange. If you can trigger it with http logging enabled
(you'd need a debug build), or reliably reproduce it, then please file a new bug.
(Reporter)

Comment 38

17 years ago
The first bit (re-using timed out, but open connections) seems to go with this
bug.

For the second bit, I can reproduce it by:
  1) load a page (I use http://www.mozillazine.org for this test)
  2) wait for it to complete loading
  3) hit reload.

When using a http/1.1 proxy, the page itself loads fine, then mozilla sends
requests for (most of) the images.  On the url above, the proxy returns 304 (not
modified) for all images.  There are a handful of images on the page though that
mozilla does *not* send new requests for.  Each of these is *not* redrawn, and
mozilla hangs (the throbber never stops spinning).  The set of images it
"forgets" to request is the same each time (tile.jpg, grass.gif, a few
others...)  I then hit 'esc' or the stop button to stop the throbber. 
Subsequently, mozilla won't load any pages.  Hitting bookmarks or typing
something into the url bar doesn't generate new requests to the proxy.  (I'm
watching the network traffic with ngrep -- this can't be a proxy bug)

This doesn't happen if the proxy is not enabled.

I'm going to file a new bug for this second one, unless someone tells me
otherwise.
(Assignee)

Comment 39

17 years ago
Checked in on 092 branch.
Status: ASSIGNED → RESOLVED
Last Resolved: 17 years ago
Keywords: vbranch
Resolution: --- → FIXED
Whiteboard: checked in on trunk → [topembed+]

Comment 40

16 years ago
Verified.
Status: RESOLVED → VERIFIED
QA Contact: benc → junruh
You need to log in before you can comment on or make changes to this bug.