Last Comment Bug 423377 - (wedidntstartthefire) Change max-persistent-connections-per-server to 6.
(wedidntstartthefire)
: Change max-persistent-connections-per-server to 6.
Status: RESOLVED FIXED
: perf
Product: Core
Classification: Components
Component: Networking: HTTP (show other bugs)
: Trunk
: All All
: P1 enhancement (vote)
: ---
Assigned To: Robert Sayre
:
Mentors:
Depends on: 448022 458545
Blocks:
  Show dependency treegraph
 
Reported: 2008-03-16 22:41 PDT by Robert Sayre
Modified: 2009-04-02 20:43 PDT (History)
43 users (show)
ted: blocking1.9+
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
prefs (2.23 KB, patch)
2008-03-17 22:16 PDT, Robert Sayre
bzbarsky: review+
cbiesinger: review+
brendan: superreview+
Details | Diff | Review

Description Robert Sayre 2008-03-16 22:41:30 PDT
And in my totally unscientific tests, it rocks!
Comment 1 Boris Zbarsky [:bz] (Out June 25-July 6) 2008-03-16 22:48:14 PDT
Do we want to bump some of the other max settings in the process?
Comment 2 Gen Kanai [:gen] 2008-03-16 23:12:56 PDT
As mentioned by Kohei in bug#418962#c3 Fasterfox tweaks these settings:

user_pref("network.http.max-connections", 40);
user_pref("network.http.max-connections-per-server", 16);
user_pref("network.http.max-persistent-connections-per-proxy", 12);
user_pref("network.http.max-persistent-connections-per-server", 6);
user_pref("nglayout.initialpaint.delay", 0);
Comment 3 Mike Shaver (:shaver -- probably not reading bugmail closely) 2008-03-17 06:12:02 PDT
Those settings would get us banned from the internet.  Let's deal with other settings in other bugs, so that the known-wanted tweaks don't get held up behind research.
Comment 4 Mike Beltzner [:beltzner, not reading bugmail] 2008-03-17 06:25:42 PDT
Let's not scope creep, here, and keep this bug about max-persistent-connections.

From dev-apps-firefox:

Firefox 2: 2
Opera 9.26: 4
Opera 9.5 beta: 4
Safari 3.0.4 Mac/Windows: 4
IE 7: 2
IE 8: 6

Based on that I'm wondering if we should be conservative and just double it to 4. Rob, does that still rock?
Comment 5 Ted Mielczarek [:ted.mielczarek] 2008-03-17 08:03:26 PDT
Blocking on resolution one way or the other. (Personally, I think we're safe doing whatever IE8 does, although it's hard to say if that's what they'll ship in their final version or not.)
Comment 6 Ted Mielczarek [:ted.mielczarek] 2008-03-17 08:04:53 PDT
-> sayrer (this is just a pref tweak AIUI)
Comment 7 Mike Shaver (:shaver -- probably not reading bugmail closely) 2008-03-17 08:06:17 PDT
They also provide a scriptable interface to give sites some more control, which we won't be aping.  We can reduce the default in a .0.x patch if all hell breaks loose, but I think it rather unlikely; app developers seem pretty universally in favour of an increased limit.
Comment 8 Boris Zbarsky [:bz] (Out June 25-July 6) 2008-03-17 09:38:52 PDT
Limiting this to just max-persistent-connections without raising network.http.max-connections or network.http.max-connections-per-server may not make much sense.  If we're allowing 6 persistent connections but only 8 connections total, that seems odd to me.  As I understand, IE upped both limits.

The current numbers in all.js are:

pref("network.http.max-connections", 24);
pref("network.http.max-connections-per-server", 8)
pref("network.http.max-persistent-connections-per-server", 2);
pref("network.http.max-persistent-connections-per-proxy", 4);

So it looks like Fasterfox triples the two smallest numbers, doubles the 8, and bumps up the 24 "some".

I would advocate doing at least:

pref("network.http.max-connections", 24);
pref("network.http.max-connections-per-server", 12)
pref("network.http.max-persistent-connections-per-server", 4);
pref("network.http.max-persistent-connections-per-proxy", 8);

I think we could even go to 6 on max-persistent-connections with max-connections at 12, though at that point 16 might make more sense.
Comment 9 Eric H. Jung 2008-03-17 11:05:08 PDT
(In reply to comment #8)

> pref("network.http.max-persistent-connections-per-server", 4);
> pref("network.http.max-persistent-connections-per-proxy", 8);

Why would you make these different?
Comment 10 Boris Zbarsky [:bz] (Out June 25-July 6) 2008-03-17 11:40:43 PDT
Because if you're being proxied then _all_ your network requests are going to the proxy.  If you browse multiple sites, tough.  You're limited by the proxy cap, no questions asked.
Comment 11 Eric H. Jung 2008-03-17 11:58:27 PDT
(In reply to comment #10)
> Because if you're being proxied then _all_ your network requests are going to
> the proxy. 

Not if you're using the FoxyProxy extension, but point taken.
Comment 12 Boris Zbarsky [:bz] (Out June 25-July 6) 2008-03-17 12:03:18 PDT
Sure.  Or you could be using a PAC, or whatever.  But in many cases (e.g. corporate setups) the user is just being proxied, period.
Comment 13 Sylvain Pasche 2008-03-17 13:00:54 PDT
An interesting article about Testing IE8’s Connection Parallelism:
http://www.ajaxperformance.com/2008/03/16/testing-ie8s-connection-parallelism/

Main points:
1) When using connection parallelism tricks (spreading resources on
image1.example.com, image2.example.com, image2.example.com all pointing to your
server), the performance can be reduced by the large number of connections
opened (n_hosts * n_connections_per_host).

2) IE 8 is apparently going to handle limits differently for dial-up: "Kris Zyp
and Steve Souders have pointed out that IE8 will use 2 connections per host for
dial-up users."

I think that authors should be taught that doing 1) could be harmful if they do
not take into account the default connection settings of the browsers they are
dealing with (yay for UA sniffing).
2) would be a good enhancement for dial-up users. But I guess they are a
minority these days.
Comment 14 Robert O'Callahan (:roc) (Exited; email my personal email if necessary) 2008-03-17 14:06:19 PDT
(In reply to comment #7)
> They also provide a scriptable interface to give sites some more control, which
> we won't be aping.

As far as I could tell from looking at the whitepaper, it's a read-only property. So sites can use it to throttle their XHRs etc but they can't change it.
Comment 15 Robert Sayre 2008-03-17 22:16:12 PDT
Created attachment 310171 [details] [diff] [review]
prefs

The thinking here is this: our limits are antiquated. Sites are demanding more connections due to JS traffic, and we're using up more of the total connection limit with suggest requests, antimalware, extension traffic, etc.
Comment 16 Boris Zbarsky [:bz] (Out June 25-July 6) 2008-03-17 22:20:22 PDT
Comment on attachment 310171 [details] [diff] [review]
prefs

Alea iacta est.  Or at least sorta.
Comment 17 Brendan Eich [:brendan] 2008-03-17 22:23:31 PDT
Comment on attachment 310171 [details] [diff] [review]
prefs

Sure, let's try this for b5.

/be
Comment 18 Boris Zbarsky [:bz] (Out June 25-July 6) 2008-03-18 07:30:37 PDT
This didn't seem to affect Tp, right?  Connection too fast and too low-latency there to notice?
Comment 19 Ted Mielczarek [:ted.mielczarek] 2008-03-18 07:34:31 PDT
Obviously we need some sort of Tgmail.
Comment 20 Mike Hommey [:glandium] 2008-03-18 12:48:57 PDT
This change might induce problems with Apache servers. Each connection kept alive on an apache server takes a "processing unit" (be it a thread or a process), which means that taking 6 connections instead of 2 will multiply by 3 the number of processing units used on servers. Except for an experimental apache engine, there is no pooling of kept alive connections, so each kept alive connection prevents a processing unit to give content to other browsers.
And considering it is still recommended to use the prefork mpm (where processing units are processes) for php (because it is not thread safe), increasing the number of processes may not be an option.
Surely, anyone can easily DoS a web site running apache by opening a lot of keep-alive connections, but it would be better if genuine browsers wouldn't do it.
Comment 21 Robert Sayre 2008-03-18 13:03:04 PDT
(In reply to comment #20)
> This change might induce problems with Apache servers.

Surely such problems would have already been observed with Safari 3 and Opera 9 (which both allow 4 persistent connections). Betas are the best place to try these changes.

Comment 22 Boris Zbarsky [:bz] (Out June 25-July 6) 2008-03-18 13:07:11 PDT
> will multiply by 3 the number of processing units used on servers.

I'd love to hear your reasoning for that conclusion.  Here's why I think it's not quite the right conclusion.

What will be multiplied by 3 is the number of simultaneous persistent connections a given user can hold open to the server.  If we assume that this change doesn't affect the number of users, the average number of total connections held open on the server will be scaled by the product of two ratios: The ration of the average number of connections held open to the server after this change to the number held open before this change and the ratio of average time a connection is held open after this change to the average time a connection is held open before this change.

During pageload, either the loads will be parallelized over more connections (with pageload being faster as a result), or they won't be (and then you don't have more connections).  In either case, I would expect the product to be close to unity for this scenario.  The client will hold more connections open to the server, perhaps, but hold them open for less time.  There is an obvious limiting factor here, which is network bandwidth: if the user can't actually benefit speed-wise from more connections, then the server loses.  There are plans to dynamically lower the limit on dial-up connections.

Similar arguments would apply to pretty much every workload that I can realistically think of where time-averaging is applicable.  The only exceptions I've been able to think of are:

1)  A server which has all its users hitting it all at once (download after
    some announcement, presumably, since page loads are too transient for this
    to happen on any large scale).
2)  A server to which users connect and then suck down arbitrary amounts of
    data using as many connections as they can, with the connection lifetimes
    fixed.  I'm not sure why this would happen.  The users would be getting 3
    times the data they get now (again, assuming bandwidth is not the limiter),
    so I'd expect them to be connecting 3 times less often, on average, in any
    sane situation.

There's also the minor issue that every single other browser has the limit higher than 2, so Apache loses no matter what we do.
Comment 23 Mike Hommey [:glandium] 2008-03-18 13:25:19 PDT
(In reply to comment #22)
> The client will hold more connections open to the
> server, perhaps, but hold them open for less time.

Keep-alive usually holds the connections much longer than download time. Holding 6 connections alive is going to keep 6 connections open for the keep-alive timeout (15 seconds by default on Apache), while it would have been 2 before. Or do you mean keep-alive is disabled ?
Comment 24 Mike Hommey [:glandium] 2008-03-18 13:28:01 PDT
(In reply to comment #21)
> (In reply to comment #20)
> > This change might induce problems with Apache servers.
> 
> Surely such problems would have already been observed with Safari 3 and Opera 9
> (which both allow 4 persistent connections). Betas are the best place to try
> these changes.

How much "market share" do Safari 3 and Opera 9 have ? How likely is it that there are enough Safari 3 and Opera 9 users on a same server that it would be a big difference ? Same applies with betas. You won't know how broken your choice was until Apache admins are forced to disable keep-alive on their servers.
Comment 25 Boris Zbarsky [:bz] (Out June 25-July 6) 2008-03-18 13:29:59 PDT
No, keep-alive is enabled (otherwise we wouldn't be talking about persistent connections).  So in the above calculations, the right ratio is
(lifetime + 15s)/(lifetime + 15s) instead of lifetime/lifetime.  This does skew the results somewhat, though I'd be interested in what typical lifetimes look like.  We're still not likely to be saturating the 6 connections on a lot of pages (due to blocking on scripts), and Apache still has this issue no matter what we do.

As for the rest, so far the plan is to ship this in a beta.  Then we'll see.
Comment 26 Boris Zbarsky [:bz] (Out June 25-July 6) 2008-03-18 13:30:38 PDT
Of course if there is hard data that this causes problems in the beta, we'd love to hear about it!
Comment 27 Mike Hommey [:glandium] 2008-03-18 13:45:52 PDT
If you want to be safe, you'd better close connections when there are several waiting for keep-alive timeout on the same server. You're not really likely to require all of them during this time. You're usually not even using a single one. Anyways, one should be enough.
Comment 28 Mike Hommey [:glandium] 2008-03-18 13:46:19 PDT
(better safe than sorry, because sorry would be too late)
Comment 29 Steve Parkinson 2008-03-18 23:49:54 PDT
I am surprised no one has mentioned section 8.1.4 of the HTTP 1.1 RFC:

http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8

quote: "A single-user client SHOULD NOT maintain more than 2 connections with any server or proxy."
Comment 30 Robert Sayre 2008-03-18 23:53:38 PDT
(In reply to comment #29)
> I am surprised no one has mentioned section 8.1.4 of the HTTP 1.1 RFC:

I've seen the document once or twice. Which browsers comply with the section you've quoted?
Comment 31 Steve Parkinson 2008-03-19 00:03:53 PDT
(In reply to comment #30)
> I've seen the document once or twice. Which browsers comply with the section
> you've quoted?
> 

I think mike already answered that in comment #4. 

I do think it's worth documenting in this bug where the '2 connections' originally came from, and why it's been like that for so long.

Since the RFC says SHOULD and not MUST, that leaves some leeway to increase it.

Comment 32 Kevin Hamilton 2008-03-19 14:20:34 PDT
(In reply to comment #26)
> Of course if there is hard data that this causes problems in the beta, we'd
> love to hear about it!
> 

Are there any identified areas which are likely to see high FF3b5 user percentages?  Can anyone reach out to them and see if they can report before/after performance?  I'm thinking, for example, AMO, mozillazine.org, planet.m.o?
Comment 33 Yusuf Goolamabbas 2008-03-21 19:10:41 PDT
(In reply to comment #20)
> the number of processing units used on servers. Except for an experimental
> apache engine, there is no pooling of kept alive connections, so each kept
> alive connection prevents a processing unit to give content to other browsers.
> And considering it is still recommended to use the prefork mpm (where
> processing units are processes) for php (because it is not thread safe),
> increasing the number of processes may not be an option.
> Surely, anyone can easily DoS a web site running apache by opening a lot of
> keep-alive connections, but it would be better if genuine browsers wouldn't do
> it.
> 

Running a reverse-proxy (Squid) or todays favourite's (nginx,Varnish, lighttpd 1.5) in front of Apache has always been best practice. Let Apache listen on 127.0.0.1, turn Keep-Alive Off and front it with nginx who does all the keepalive management
Comment 34 chris hofmann 2008-11-27 18:36:56 PST
moving up from 3.0.4 to running the 3.1b2 candidate builds in the last week I've seen a lot of strange network behavior with attempts to visit sites timing out.  then hit reload once or twice and the site loads ok.  All very hard to preproduce in any way.  I don't know if it is flakey connections at home or work, or what it might be.  Is there any chance that upping the connections might create race conditions or logjams in network loading?   I'll keep trying to see if the is a way to reproduce something, but we should watch for this kind of feedback from others, and if we do figure out what changes it might be connected to.
Comment 35 Boris Zbarsky [:bz] (Out June 25-July 6) 2008-11-27 19:42:57 PST
Chris, you could try setting these prefs to the old values locally and seeing whether that has an effect....
Comment 36 Ted Mielczarek [:ted.mielczarek] 2008-11-28 18:26:11 PST
Also, FWIW, we fixed this in Firefox 3.0, so there should be no difference between that and 3.1b2.
Comment 37 chris hofmann 2008-12-10 17:40:52 PST
re: last three comments.

I changed back to older connection levels and didn't see any change in behavior.  Still lots of annoying time-outs. I also changed back to 3.0.4 and don't seem to see the problem there.   

One specfic place I saw a high pct. of problems were with 3.1b2 was when I clicking on links to wikipedia pages from google search results pages.  I also noticed that the delay might be early in the cycle of trying to connect to a site, and status bar messages that indicate it might be in the stages of DNS lookups.   Maybe bugs 453403 and 464838 might be behind my problem.
Comment 38 Shawn Wilsher :sdwilsh 2008-12-10 17:47:03 PST
(In reply to comment #37)
> One specfic place I saw a high pct. of problems were with 3.1b2 was when I
> clicking on links to wikipedia pages from google search results pages.  I also
> noticed that the delay might be early in the cycle of trying to connect to a
> site, and status bar messages that indicate it might be in the stages of DNS
> lookups.   Maybe bugs 453403 and 464838 might be behind my problem.
Are you maybe seeing bug 467562?
Comment 39 Frank Schroeder 2009-03-31 14:25:16 PDT
I'm seeing another problem here with this setting. I've got a small embedded device - a VoIP phone Siemens Gigaset S685 - which just gets flooded by FF 3.1b3 with requests. By setting network.http.max-connections-per-server to 15 or even to 8 the web ui of the device becomes unusable. Only setting it to 4 makes the web ui usable again. 

If I understand the comments correctly bumping the number of concurrent connections per server should improve sites which use a lot of different JS scripts. But doesn't changing the browsers networking behavior "fix" this problem in the wrong place? Shouldn't the app developers combine and compress their JS scripts to a single file so that the browser doesn't have to load lots of JS files?

In any case setting this value so high breaks other applications to the point where they are unusable and not easy to fix. 

Running FF 3.1b3 on a Mac.
Comment 40 Boris Zbarsky [:bz] (Out June 25-July 6) 2009-04-02 20:43:11 PDT
> If I understand the comments correctly bumping the number of concurrent
> connections per server should improve sites which use a lot of different JS

I don't think you understand correctly.  The goal is to improve any site which uses more than the previous max-connections-per-server number of resources on the page (images, stylesheets, etc).  Note that the value was 8 for a very very long time, so if that value doesn't work with your device then it's not like this patch made things any worse.  Note that all modern browsers use values that are at least this high.

It really sounds like your real problem is a severely buggy HTTP server on the device in question...

Note You need to log in before you can comment on or make changes to this bug.