987895 - A/B test SPDY

Status: RESOLVED FIXED

(Cloud Services :: Operations: Marketplace, task, P2)

Description

[Wil Clouser [:clouserw]]

There has been a lot of talk about SPDY but we have no hard numbers to say whether it will greatly improve the experience or not.  Let's get some A/B numbers so we know whether it's worth pursuing or not.  We should do this on a normal network connection and also over EDGE on the phones.

Comment 2

[Wil Clouser [:clouserw]]

This isn't A/B since it's -dev/prod but should give an idea:

-dev:  http://www.webpagetest.org/result/140327_4A_156F/
prod: http://www.webpagetest.org/result/140327_QR_157B/

The waterfall is pretty interesting.  If you look under it at the connections you can see on prod it has to make a half dozen connections to the CDN to get all the images, but only one with SPDY.  I suspect we'll see a performance improvement if we turn it on, particularly because our SSL negotiations are so slow (~70ms *inside* the US from what I can tell).  If we turn it off on -dev and retest we can get truer numbers but I think it's worth pursuing at this point.

Comment 3

[Jeremy Orem [:oremj]]

Attachment: marketplace.allizom.org.har - stage no SPDY

Comment 4

[Jeremy Orem [:oremj]]

Attachment: marketplace-dev.allizom.org.har - with SPDY

Comment 5

[Jeremy Orem [:oremj]]

Attachment: marketplace-dev.allizom.org.har - with SPDY

Comment 6

[Jeremy Orem [:oremj]]

Attachment: marketplace.allizom.org.har - stage no SPDY

Comment 7

[Wil Clouser [:clouserw]]

These are interesting numbers, although admittedly unexpected.  Jeremy says dev and stage are both on similar hardware so in theory they should be a good A/B test, but results from -dev seem substantially slower.  Some callouts:

* For the initial search, -dev takes 30% longer.  If I'm reading the graphs right that time is all in the connecting/handshaking phase.  If you look at most requests it seems like the slowness is in the handshaking/connecting stage, which SPDY should greatly reduce.

* On -dev there were 54 requests, on stage there were 73 - it looks like all of the images weren't loaded on -dev.  This is likely due to ngoke's lazy-loading patch.

* Sizes don't line up.  On stage include.js is 15k larger, includes.css is 5k larger.  When we're talking about 5k/sec those will add up. We should test when dev and stage are closer together since cvan has been optimizing.

* <offtopic> We load 4 different typefaces of the firansansot font.  I wonder if we need all of them?

I'd suggest we retest after a fresh tag instead of while stage is frozen.

Comment 8

[Jeremy Orem [:oremj]]

Attachment: marketplace.allizom.org.har - stage no SPDY

Comment 9

[Jeremy Orem [:oremj]]

Attachment: marketplace-spdy.allizom.org.har - stage with SPDY

Comment 10

[Doug Turner (:dougt)]

pat, can you work with wil to figure out why spdy isn't working well here.  It might also be that the application is just doing silly things.

Comment 11

[Jeremy Orem [:oremj]]

The pageload time with or without SPDY, over edge, seems directly proportional to bytes served.

Comment 12

[Patrick McManus [:mcmanus]]

I'm having a hard time knowing exactly what I'm looking at as I'm unfamiliar with the project. So I'm going to beg your indulgence and ask for some particular explicit data.

please provide a link to a WPT run using Firefox Nightly (available at WPT dulles) and a mobile profile of your choosing (or more than one if you like). Please include the packet capture (tcpdump) option under the advanced tab.

Repeat the process with spdy enabled/disabled as appropriate. Obviously, if the content is different comparisons will be harder :)

The WPT data I saw in earlier comments was based on chrome over cable networks, so rather than trying to sort out the differences let's just get the primary source material.

spdy's primary win is when you have high latencies and many objects that are too small to saturate the bandwidth at that latency. Larger objects will indeed be able to saturate the bandwidth and then the performance will devolve to match http/1. sometimes that is the case - often it isn't. depends on the network parameters.

I'd like to know what are your parameters for your target (is that edge? can we express that in rtt and bw? that wasn't a fully standardized term).. the total transfer on the page is something like 700KB which might simply totally bottlenecked on bandwidth.

From glancing at the first waterfalls I also wonder if we're blocking on some js and that can be worked around.. but let's get packet caps and waterfalls specifically of gecko over a mobile profile on spdy and see what we're looking at.

Comment 13

[Patrick McManus [:mcmanus]]

fwiw - wikipedia describes edge as gprs+ and ~200kbit/s of bandwidth at 600-700ms rtt (up to 1000) and https://www.cl.cam.ac.uk/research/srg/netos/coms/rc277/globe02.pdf describes 1000+ms latencies too

my old notes have something similar.

of course 700KB of data at 200kbit/s is 30 seconds in the most ideal world.. tcp/tls/http-spdy will only add to that (and various approaches jostle over how much they are adding)

what is the goal number?

Comment 14

[Jeremy Orem [:oremj]]

On firefox nightly with 300/200 Kbps with 500ms latency:

HTTP: http://www.webpagetest.org/result/140411_3T_RW2/
SPDY: http://www.webpagetest.org/result/140411_J2_RW8/


Plain HTTP still seems to be winning.

Comment 15

[Jeremy Orem [:oremj]]

Same settings with chrome (WPT seems to work a little better with chrome):

HTTP: http://www.webpagetest.org/result/140411_JS_S9Y/
SPDY: http://www.webpagetest.org/result/140411_AC_S94/

No difference in load time.

Comment 16

[Jeremy Orem [:oremj]]

One more test with 2000ms latency:

SPDY: http://www.webpagetest.org/result/140411_19_SC2/
HTTP: http://www.webpagetest.org/result/140411_1B_SC7/

Still no difference.

Comment 17

[Matt Basta [:basta]]

(In reply to Jeremy Orem [:oremj] from comment #16)
> Still no difference.

That's not strictly true. The page load event doesn't finish until about the same time, but that's because it's blocked on Persona (which doesn't use SPDY). Native persona means (or should mean that) those requests won't be present on-device.

If you consider the waterfalls sans Persona, the SPDY marketplace finishes loading 10s sooner.

Also note that the tests are slightly flawed in-so-far as some of the connections are not properly using SPDY. In the results from comment #16, you can see that lines 4 and 26 are creating independent connections. If SPDY were working as intended for those connections, you'd potentially see an extra 4-6 second improvement.

Another thing to note is that in the SPDY version, something is visible on the screen (as-is) four seconds earlier than in the non-SPDY version.

Comment 18

[Patrick McManus [:mcmanus]]

(In reply to Jeremy Orem [:oremj] from comment #14)
> On firefox nightly with 300/200 Kbps with 500ms latency:
> 
> HTTP: http://www.webpagetest.org/result/140411_3T_RW2/
> SPDY: http://www.webpagetest.org/result/140411_J2_RW8/
> 
> 

I only really looked at this dataset - it seemed closest to comment 13.. feel free to direct me to something else if its of particular interest.

unfortunately, I think you need to rerun beacuse the two tests don't seem to have even close to the same content.

The http/1 case is 86 requests and 752KB of content, the spdy case is 36 requests and 1,079KB. Can't compare those meaningfully.

Both cases are getting killed by OCSP though. easily 15 seconds in the spdy case.. also bad, but less clear in http/1 case.

You can fix this with OCSP stapling on the server. I would make that a top priority. Also make sure the server is running IW10, that's pretty critical to proper spdy performance.

let me know if you need help following up on either of those.

Comment 19

[Jeremy Orem [:oremj]]

(In reply to Patrick McManus [:mcmanus] from comment #18)
> (In reply to Jeremy Orem [:oremj] from comment #14)
> > On firefox nightly with 300/200 Kbps with 500ms latency:
> > 
> > HTTP: http://www.webpagetest.org/result/140411_3T_RW2/
> > SPDY: http://www.webpagetest.org/result/140411_J2_RW8/
> > 
> > 
> 
> I only really looked at this dataset - it seemed closest to comment 13..
> feel free to direct me to something else if its of particular interest.
> 
> unfortunately, I think you need to rerun beacuse the two tests don't seem to
> have even close to the same content.
> 
I think something is wrong with WPT and firefox. The results only seem accurate when choosing chrome.

Comment 20

[Jeremy Orem [:oremj]]

That 2000ms test was a little unrealistic. Here is a new test with 600ms latency, which is more inline with the real world.

SPDY: http://www.webpagetest.org/result/140411_TP_WX2/
HTTP: http://www.webpagetest.org/result/140411_KX_WWY/

In this test, msFirstPaint, domContentLoaded, and loadEvent are all within 2 seconds difference.

Comment 21

[Patrick McManus [:mcmanus]]

(In reply to Jeremy Orem [:oremj] from comment #20)
> That 2000ms test was a little unrealistic. Here is a new test with 600ms
> latency, which is more inline with the real world.
> 
> SPDY: http://www.webpagetest.org/result/140411_TP_WX2/


WPT is indeed being weird. It doesn't even have the firefox-injected spdy response header marker in there for marketplace-stage, though the connection count certainly suggests it is using it.

my initial suggestions on how to make it go faster

1] For whatever reason WPT is not showing OCSP (potentially because OCSP doesn't impact the DOM and doesn't show up in resource timing) - those runs don't show OCSP directly but I just confirmed with my desktop that the device under test isn't doing OCSP stapling so it should be reflected. And we've seen OCSP is a giant bottleneck for you (as it always is) - so definitely pursue stapling as a work item. OCSP without stapling is going to cost you an absolute minimum of 3 RTT of blocking time each time you connect for the first time to a new hostname. So that's 1800ms x 3 or 4 here.. and its often a lot worse.

2] confirm you've got IW10 on every server. Even the marketplace-spdy would benefit from it, but marketplace-stage would really benefit from it. without the tcpdump (which wasn't included) I can't tell if it was doing IW10 or not - but when you hit the CDN with a blast of parallel requests it needs this to be able to respond effectively to them.

3] you can improve performance if marketplace-spdy.allizom.org and marketplace-stage.mozflare.net can be brought under the same host - you basically block for 2 seconds on this switchover. If that can't happen then if marketplace-stage has a stable IP, publish an A record for it as marketplace-stage.allizom.org - keeping the domain structure consistent will save recursive DNS lookups. don't use cnames - they add rtts. and rtts kill you at 500ms rtt times :) (you do have the meta dns prefetch which is cool, but its after the css reference has already been seen so its pretty much moot at that point)

4] make marketplace-spdy also run spdy (right now its just on the CDN). Its only serving one or two resources so this is really for use as a signalling mechanism that it is ok for firefox to do TLS false start with your server. We need a signal to work around old buggy https servers, so even though it is not really a spdy optimization it will help. This will save you a round trip. They add up! If you really don't want to run spdy, its sufficient to have the server negotiate http/1 with npn or alpn instead.

5] it isn't reflected in the WPT log, (again probably like ocsp because it doesn't hit the dom) but when I access this site on desktop and look at it with devtools I always see https://marketplace-spdy.allizom.org/services/csp/policy?build=6767 as the second transaction - and it strictly blocks connecting to the CDN.  It must come from X-Content-Security-Policy-Report-Only: policy-uri /services/csp/policy?build=6767 which is a response header in slash.
  a] thanks for using CSP!
  b] use Content-blah instead of X-Content-blah
  c] can the policy be inlined into the header instead of done by reference like that? its costing at least a round trip of blocking time before the cdn can be contacted.. maybe more since it is not running spdy and has to fight with the html download so may need a new connection

6] starting at request 44 there is a run of images all requested together that have obviously been waiting for something to complete.. I bet that wait time can be moved up. how does include.js make its requests? i.e. does it try and do some kind of smart scheduling or does it go ahead and make the image requests as fast as it can? with spdy it should do the latter.

7] include.js is clearly a bottleneck for you. It seems to load most of the other resources (sometimes indirectly?), but that can't happen until the script is loaded completely... and it appears to have a lot of stuff that is extraneous to the frontpage on it.. credits.html for example is inlined in it. it seems you would be much better off loading that stuff separately.

Comment 22

[Patrick McManus [:mcmanus]]

(In reply to Jeremy Orem [:oremj] from comment #19)

> I think something is wrong with WPT and firefox. The results only seem
> accurate when choosing chrome.

the wpt owner tells me wpt support for spdy isn't up to snuff yet (maybe firefox specific in wpt). so that explains that - sadly.

If you're using os x desktop, I've heard nice things about http://mattgemmell.com/network-link-conditioner-in-lion/ combined with the firefox dev tools.

Comment 23

[Jeremy Orem [:oremj]]

Yeah, the network link conditioner is what I used in my initial tests.

Comment 24

[Jeremy Orem [:oremj]]

(In reply to Patrick McManus [:mcmanus] from comment #21)
> (In reply to Jeremy Orem [:oremj] from comment #20)
> > That 2000ms test was a little unrealistic. Here is a new test with 600ms
> > latency, which is more inline with the real world.
> > 
> > SPDY: http://www.webpagetest.org/result/140411_TP_WX2/
> 
> 
> WPT is indeed being weird. It doesn't even have the firefox-injected spdy
> response header marker in there for marketplace-stage, though the connection
> count certainly suggests it is using it.
> 
> my initial suggestions on how to make it go faster
> 

> 
> 2] confirm you've got IW10 on every server. Even the marketplace-spdy would
> benefit from it, but marketplace-stage would really benefit from it. without
> the tcpdump (which wasn't included) I can't tell if it was doing IW10 or not
> - but when you hit the CDN with a blast of parallel requests it needs this
> to be able to respond effectively to them.
We can adjust this on marketplace-spdy, via a load balancer tweak, but we have no control over CDN tcp settings.


> 3] you can improve performance if marketplace-spdy.allizom.org and
> marketplace-stage.mozflare.net can be brought under the same host - you
> basically block for 2 seconds on this switchover. If that can't happen then
> if marketplace-stage has a stable IP, publish an A record for it as
> marketplace-stage.allizom.org - keeping the domain structure consistent will
> save recursive DNS lookups. don't use cnames - they add rtts. and rtts kill
> you at 500ms rtt times :) (you do have the meta dns prefetch which is cool,
> but its after the css reference has already been seen so its pretty much
> moot at that point)
marketplace-stage is a CDN endpoint, so an A record is not possible.
> 
> 4] make marketplace-spdy also run spdy (right now its just on the CDN). Its
> only serving one or two resources so this is really for use as a signalling
> mechanism that it is ok for firefox to do TLS false start with your server.
> We need a signal to work around old buggy https servers, so even though it
> is not really a spdy optimization it will help. This will save you a round
> trip. They add up! If you really don't want to run spdy, its sufficient to
> have the server negotiate http/1 with npn or alpn instead.
The load balancer in front of marketplace-spdy hasn't implemented SPDY support.

> 
> 6] starting at request 44 there is a run of images all requested together
> that have obviously been waiting for something to complete.. I bet that wait
> time can be moved up. how does include.js make its requests? i.e. does it
> try and do some kind of smart scheduling or does it go ahead and make the
> image requests as fast as it can? with spdy it should do the latter.
> 
> 7] include.js is clearly a bottleneck for you. It seems to load most of the
> other resources (sometimes indirectly?), but that can't happen until the
> script is loaded completely... and it appears to have a lot of stuff that is
> extraneous to the frontpage on it.. credits.html for example is inlined in
> it. it seems you would be much better off loading that stuff separately.

A developer should be able to comment on these last two suggestions.

Comment 25

[Wil Clouser [:clouserw]]

> > 6] starting at request 44 there is a run of images all requested together
> > that have obviously been waiting for something to complete.. I bet that wait
> > time can be moved up. how does include.js make its requests? i.e. does it
> > try and do some kind of smart scheduling or does it go ahead and make the
> > image requests as fast as it can? with spdy it should do the latter.
> > 
> > 7] include.js is clearly a bottleneck for you. It seems to load most of the
> > other resources (sometimes indirectly?), but that can't happen until the
> > script is loaded completely... and it appears to have a lot of stuff that is
> > extraneous to the frontpage on it.. credits.html for example is inlined in
> > it. it seems you would be much better off loading that stuff separately.
> 
> A developer should be able to comment on these last two suggestions.

I've filed bug 1001665 and bug 1001666 for those two suggestions and to not distract this bug. Thanks.

Comment 26

[Jeremy Orem [:oremj]]

Anything else needed to close out this bug?

Comment 27

[Wil Clouser [:clouserw]]

I filed separate bugs for the development changes.  I think it's just addressing anything else from Patrick's comment and coming to a conclusion on SPDY.

Comment 28

[Jeremy Orem [:oremj]]

IW10 has been enabled on marketplace.firefox.com

Comment 29

[Jeremy Orem [:oremj]]

Filed bug 1011152 for OCSP stapling.

Comment 30

[Jeremy Orem [:oremj]]

Closing this out. bug 871683 is still open for turning on SPDY.

Summary:

Using a SPDY enabled CDN, I wasn't able to show a significant page load speed up.

Comment 31

[Christopher Van Wiemeersch [:cvan]]

Does https://marketplace-spdy.allizom.org/ still has SPDY enabled? It appears not to me, but should it?

(In reply to Jeremy Orem [:oremj] from comment #24)
> (In reply to Patrick McManus [:mcmanus] from comment #21)
> > (In reply to Jeremy Orem [:oremj] from comment #20)
> > 
> > 2] confirm you've got IW10 on every server. Even the marketplace-spdy would
> > benefit from it, but marketplace-stage would really benefit from it. without
> > the tcpdump (which wasn't included) I can't tell if it was doing IW10 or not
> > - but when you hit the CDN with a blast of parallel requests it needs this
> > to be able to respond effectively to them.
> We can adjust this on marketplace-spdy, via a load balancer tweak, but we
> have no control over CDN tcp settings.

Which congestion window are we at right now? We should definitely be using IW10 to fit more in the initial roundtrip, otherwise we're going to get exponentially more roundtrips.

> > 4] make marketplace-spdy also run spdy (right now its just on the CDN). Its
> > only serving one or two resources so this is really for use as a signalling
> > mechanism that it is ok for firefox to do TLS false start with your server.
> > We need a signal to work around old buggy https servers, so even though it
> > is not really a spdy optimization it will help. This will save you a round
> > trip. They add up! If you really don't want to run spdy, its sufficient to
> > have the server negotiate http/1 with npn or alpn instead.
> The load balancer in front of marketplace-spdy hasn't implemented SPDY
> support.

Wouldn't the load balancer need to speak SPDY to the client since the client is the one negotiating the SSL connection?

"If the SSL connection negotiated NPN, then Chrome will speak SPDY to the proxy. Otherwise, it will speak HTTP." [1]

Are we sure that all requests are in fact using the single SPDY session rather than opening new HTTP connections?

[1] http://www.chromium.org/spdy/spdy-proxy-examples

Comment 32

[Jeremy Orem [:oremj]]

(In reply to Christopher Van Wiemeersch [:cvan] from comment #31)
> Does https://marketplace-spdy.allizom.org/ still has SPDY enabled? It
> appears not to me, but should it?
The CDN domain marketplace-spdy is using (marketplace-stage.mozflare.net), has SPDY enabled. Currently, only two requests are made to the non-CDN domain.

> 
> (In reply to Jeremy Orem [:oremj] from comment #24)
> > (In reply to Patrick McManus [:mcmanus] from comment #21)
> > > (In reply to Jeremy Orem [:oremj] from comment #20)
> > > 
> > > 2] confirm you've got IW10 on every server. Even the marketplace-spdy would
> > > benefit from it, but marketplace-stage would really benefit from it. without
> > > the tcpdump (which wasn't included) I can't tell if it was doing IW10 or not
> > > - but when you hit the CDN with a blast of parallel requests it needs this
> > > to be able to respond effectively to them.
> > We can adjust this on marketplace-spdy, via a load balancer tweak, but we
> > have no control over CDN tcp settings.
> 
> Which congestion window are we at right now? We should definitely be using
> IW10 to fit more in the initial roundtrip, otherwise we're going to get
> exponentially more roundtrips.
We have IW10 enabled on marketplace.firefox.com, however, we have no control over TCP settings that Akamai uses. Since their business is speedy delivery, I assume their TCP stack is tuned.

> 
> > > 4] make marketplace-spdy also run spdy (right now its just on the CDN). Its
> > > only serving one or two resources so this is really for use as a signalling
> > > mechanism that it is ok for firefox to do TLS false start with your server.
> > > We need a signal to work around old buggy https servers, so even though it
> > > is not really a spdy optimization it will help. This will save you a round
> > > trip. They add up! If you really don't want to run spdy, its sufficient to
> > > have the server negotiate http/1 with npn or alpn instead.
> > The load balancer in front of marketplace-spdy hasn't implemented SPDY
> > support.
> 
Unfortunately, our options right now are Riverbed Stingray and Netscaler (currently in use). Neither of them have good NPN support and they don't seem very interested in implementing it. With ALPN on the horizon, I'm not sure implementation will happen. 

> Wouldn't the load balancer need to speak SPDY to the client since the client
> is the one negotiating the SSL connection?
> 
> "If the SSL connection negotiated NPN, then Chrome will speak SPDY to the
> proxy. Otherwise, it will speak HTTP." [1]
> 
> Are we sure that all requests are in fact using the single SPDY session
> rather than opening new HTTP connections?
> 
> [1] http://www.chromium.org/spdy/spdy-proxy-examples