1135104 - windows upload speed limited to 13 megabit/s sendbuffer

Status: RESOLVED FIXED

(Core :: Networking: HTTP, defect)

Description

[Patrick McManus [:mcmanus]]

https://code.google.com/p/chromium/issues/detail?id=458685 is a chrome bug about spdy and quic upload rates on high bandwidth networks using Windows uploading a large file to google docs.

buried in that bug is a comment that they tried firefox and got only 100 MBytes in 60 seconds - that's about 13mbit/s. The RTT was ~70ms. This underperformed chrome (by how much depended on the protocol - but sometimes by 8x).

My working environment has about 6 megabit/s of uplink and I can saturate it no problem with a similar 70ms RTT, but that's not surprising if the bottleneck is 13 :)

I'm certain the issue is the size of the windows send buffer is explicitly set via pref to 128KB. the bandwidth-delay-product of a 70ms, 13mbit/s channel is about 115KB. (and that's 13mbit of goodput, so its really running faster than that on the wire over https). so that's pretty much as fast as you can go with that delay and that window.

that setting was made in bug 454990 in 2006. (I did it :)). It was in reaction to XP and the new-at-the-time vista having 8KB windows on uplink! The notes also say that some platforms (mac, linux) do autotuning, but windows only does it on the receive side. That might be outdated information - or the kind of thing we need to probe os revisions to figure out.

Anyhow - if you delete that pref (network.tcp.sendbuffer) or set it to zero (and it looks like that will take a restart) you should get the default OS behavior so we can at least figure out if things are different now. If they aren't, I guess we can consider just increasing the value. I think those tests are the most important thing.

Oh - when testing you should disable spdy/h2 as the server seems to be having some windowing issues of its own with those protocols (the reason the whole thing started) - but h1 should be fine for a "measure this firehose" test.

Comment 1

[Patrick McManus [:mcmanus]]

maybe dragana or honza has enough bandwidth and latency to test this out?

Comment 2

[Honza Bambas (:mayhemer)]

Sorry, pretty busy these days.  But I look forward to the result :)

Comment 3

[Honza Bambas (:mayhemer)]

(BTW, no way to test for me, too slow ISP at home)

Comment 4

[Dragana Damjanovic [:dragana]]

I will not have time before Monday. I will do it on Monday.

Comment 5

[u408661]

So I got about 18.5Mbit upload speed to google drive/docs in my test, but my rtt is significantly lower (23ms) than the examples above, making my BDP significantly lower than our buffer size (around 53KB for me versus 128KB for the buffer). There's no way I'm overfilling the buffer, so upping it (at least on my link in the office) won't help. I can say that disabling our window tuning did significantly reduce my upload performance (it seems to have been about 3x slower using whatever default Win7 uses), so it's quite possible that upping the buffer size will help.

:mcmanus, any preferences on how high we increase the buffer size?

Comment 6

[Patrick McManus [:mcmanus]]

so bentzel tells me chrome gets upload throughput on >= vista by doing nothing. So your report is confusing.

does 18.5 match chrome?

let's take a look at a packet cap and try and figure out what's going on with

a] our 128kb window
b] the default os windows management
c] chrome

(I think upload speed to youtube is something that is impt and pretty much everyone should do the same on, at least when using http)

Comment 7

[Dragana Damjanovic [:dragana]]

i tried it too, but i have only 45ms delay so i am hitting my connection limit at 7.5 megabit/s

Comment 8

[Dragana Damjanovic [:dragana]]

I have asked my bother to test it for me, because he has access to a link with very high upload speed at work.
And sender window size is the limit.
It uploads 200MB in 72 s with standard sender window (131072). This is around 21megabit/s and rtt is around 45sec. 
bdp is 120KB.

He increased the sender window to 4*131072 and he got 200MB in less than 20sec approximately 4 times faster.

Comment 9

[Dragana Damjanovic [:dragana]]

he also tried setting network.tcp.sendbuffer to 0 that was slower only 6megabit/s.

Comment 10

[Daniel Stenberg [:bagder]]

(In reply to Dragana Damjanovic [:dragana] from comment #8)

> I have asked my bother to test it for me, because he has access to a link
> with very high upload speed at work.

Which windows version was that done with?

Comment 11

[Dragana Damjanovic [:dragana]]

(In reply to Daniel Stenberg [:bagder] from comment #10)
> (In reply to Dragana Damjanovic [:dragana] from comment #8)
> 
> > I have asked my bother to test it for me, because he has access to a link
> > with very high upload speed at work.
> 
> Which windows version was that done with?

win 7

Comment 12

[Patrick McManus [:mcmanus]]

based on comment 8 we will make a change for >= win7

Comment 13

[Patrick McManus [:mcmanus]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=f6cce268cb79

Comment 14

[Patrick McManus [:mcmanus]]

Attachment: increase send window on modern windows

Comment 15

[Dragana Damjanovic [:dragana]]

Comment on attachment 8751214 [details] [diff] [review]
increase send window on modern windows

Review of attachment 8751214 [details] [diff] [review]:
-----------------------------------------------------------------

::: netwerk/base/nsSocketTransportService2.cpp
@@ +1197,5 @@
> +    int32_t bufferSize;
> +
> +    // If the pref is set, honor it. 0 means use OS defaults.
> +    nsresult rv = pref->GetIntPref(SEND_BUFFER_PREF, &bufferSize);
> +    if (NS_SUCCEEDED(rv)) {

Should we check got bufferSize > 0 ?

@@ +1228,1 @@
>                                          &keepaliveIdleTimeS);

alignment.

Comment 16

[Patrick McManus [:mcmanus]]

(In reply to Dragana Damjanovic [:dragana] from comment #15)
> Comment on attachment 8751214 [details] [diff] [review]
> increase send window on modern windows
> 
> Review of attachment 8751214 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: netwerk/base/nsSocketTransportService2.cpp
> @@ +1197,5 @@
> > +    int32_t bufferSize;
> > +
> > +    // If the pref is set, honor it. 0 means use OS defaults.
> > +    nsresult rv = pref->GetIntPref(SEND_BUFFER_PREF, &bufferSize);
> > +    if (NS_SUCCEEDED(rv)) {
> 
> Should we check got bufferSize > 0 ?

I want the semantic of 0 to be force "use OS default value".. nsSocketTransport2.cpp will not set the sockopt if the value is 0.

Comment 17

[Patrick McManus [:mcmanus]]

based on a test of the build from comment 14:

"GoogleDrive has too low rtt (ping rtt is 3ms) so both ff46 and your try build need only 20s

[..]tried with oneDrive
ff46 needs 4min 23s for 200MB file
and your try build only 1min 30s for the same file
It looks like it is 4 times faster that is what we wanted to see.

rtt using ping is only around 40ms, but sent me a wireshark pcap and looking at it rtt is around 135ms. So this make sense."

Comment 18

[Patrick McManus [:mcmanus]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/676c7ad5519ce78f104f0180ed225a3e9ddc2bb2
Bug 1135104 - increase send window on modern windows r=dragana

Comment 19

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/676c7ad5519c