648570 - increase total idle pconn tool to 256 (matches chrome)

Status: RESOLVED FIXED

(Core :: Networking: HTTP, defect)

Description

[Patrick McManus [:mcmanus]]

this limit is currently at 30. This limit is global, it is different than the per host limit of 6 which remains unchanged.

The primary problem with the low limit is that it forces valid idle connections to be closed in order to meet demand for connections to different hosts. This happens a lot, and of course some fraction of those connections we close would have been useful in the future. There are some pages that involve more than 5 hosts and therefore exhaust the pool all by themselves.

the resources used to manage an idle connection in the client are pretty minimal, even the kernel. 

This patch does not change the limit on mobile, the mobile pref file already overrides the 30 default.

Comment 1

[Patrick McManus [:mcmanus]]

I forgot to mention that the socket transport service may clamp the number below 256 if the OS cannot allocate that many FDs, and it also always tries to reserve 250 file descriptors for things other than sockets. (which is actually more than we reserve on OS X right now)

Comment 2

[Patrick McManus [:mcmanus]]

Attachment: raise limit to 256 v1

firefox 5 candidate

Comment 3

[Jason Duell]

Comment on attachment 524665 [details] [diff] [review]
raise limit to 256 v1

I'm going to add this comment to the patch before I land it on cedar today:

// Note: the socket transport service will clamp the number
// below 256 if the OS cannot allocate that many FDs, and it 
// also always tries to reserve 250 file descriptors for 
// things other than sockets.   

Object ASAP if you want different wording.

Comment 4

[Patrick McManus [:mcmanus]]

(In reply to comment #3)
> Comment on attachment 524665 [details] [diff] [review]
> raise limit to 256 v1
> 
> I'm going to add this comment to the patch before I land it on cedar today:
> [..] 
> 
> Object ASAP if you want different wording.

I'll add that comment and land it on cedar myself if that's ok (along with a few other).. it is going through Try right now and I want to confirm that result.

Comment 5

[Jason Duell]

Landed on cedar:

   http://hg.mozilla.org/projects/cedar/rev/33a01f08bbf3

Comment 6

[Benjamin Stover (:stechz)]

Pushed to m-c: http://hg.mozilla.org/mozilla-central/rev/33a01f08bbf3

Comment 7

[(no longer active)]

Backed out because of mobile orange.

Comment 8

[Jason Duell]

http://hg.mozilla.org/mozilla-central/rev/e84a9cf9fe6c

Comment 9

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Backed out because bug 607741 didn't compile.
http://hg.mozilla.org/mozilla-central/rev/fc1cb8500847

Comment 10

[Patrick McManus [:mcmanus]]

http://hg.mozilla.org/mozilla-central/rev/8662b2483233

Comment 11

[Cattleya]

I think this change make Firefox slower, 256 is too much, tested on this page and Firefox much slower than before.

http://stevesouders.com/hpws/max-connections.php?t=1316794119
Read this thread: http://forums.whirlpool.net.au/archive/1766023

I think 48 is a good choice, my result with 30 connection is ~ 13000ms, 256 is too slow, only 16000-18000ms, fastest is 48, from 11000ms - 8000ms.
Firefox shouldn't copy too much from Google Chrome, Opera much better than Chrome.

Comment 12

[Patrick McManus [:mcmanus]]

(In reply to Cattleya from comment #11)
> I think this change make Firefox slower, 256 is too much, tested on this
> page and Firefox much slower than before.
> 
> http://stevesouders.com/hpws/max-connections.php?t=1316794119

you can certainly over-parallelize, and that page (intentionally) does so for measurement purposes.

This bug changed the upper limit of parallelization that a server can opt into through domain sharding, which is really something that is done intentionally (as with your above example). The appropriate level of sharding depends on the nature of the content and is something content owners control.

Comment 13

[[:Cww]]

When did this make it out to Firefox release?  We saw a lot of complaints about this this past week in Firefox 7: https://support.mozilla.com/en-US/questions/881844 -- I'm blaming this bug because reverting the change fixes the issue but it's possibly tied into something else.

Comment 14

[Jason Duell]

Yes, AFAICT this was new for FF 7.

I've opened bug 692260 for the fix.

Comment 15

[Danial Horton]

alot of half baked NIC drivers can choke when you spam alot of HTTP connections, and another is that many Cheap Modems can't tolerate more than 100 connections at once, resulting in either the modem hanging or browsing to fail until the browser finishes streaming.

thats not to fail to mention that many servers are configured to BAN your ip if you attempt to hammer so many connections.

Comment 16

[Patrick McManus [:mcmanus]]

(In reply to Danial Horton from comment #15)
> alot of half baked NIC drivers can choke when you spam alot of HTTP
> connections, and another is that many Cheap Modems can't tolerate more than
> 100 connections at once, resulting in either the modem hanging or browsing
> to fail until the browser finishes streaming.

citations or references? We can get them in the lab to test them out. 

> thats not to fail to mention that many servers are configured to BAN your ip
> if you attempt to hammer so many connections.

the limit of 6 connection per server remains unchanged. The discussion here is a total connection limit.

Comment 17

[Danial Horton]

A short list from the Vuze wiki, but there are countless references on Whirlpool. With a quick search i can pull them up.

"The following routers have known problems with too many simultaneous connections. Limiting "Max connections globally" in Vuze's Transfer options to 200 or fewer should fix the problems:"

  D-Link DI-624
    D-Link DSL-G664T
    Linksys BEFSR41V4/BESR41
    Linksys WRT54G (upgrading firmware dramatically helps)
    Linksys Wireless-b
    Netgear DG632
    Netgear DG834G
    Netgear MR814
    Netgear Rangemax 802.11n WPN824
    Netgear WGT524
    Netgear WGR614
    SpeedStream 5660 in Router/NAPT configuration. Latest firmware is 2.(3).7. Alternate workarounds:
        Switch to bridged mode. (For security, firewall your network.)
        When it dies, just power-cycle the router and continue on. 
    W-Linx MB401-S (and SMC Barricade 7004 BR, which is identical in construction)
    WRT54G/GL/GS

Comment 18

[Danial Horton]

the Zyxel 600 isn't listed but i can confirm it would reset itself when it reached 240 connections.

Comment 19

[Scoobidiver (away)]

This bug is fixed. The discussion should continue in bug 692260.