903484 - SPDY: image requests to same host but different domain very slow

Status: RESOLVED FIXED

(Core :: Networking: HTTP, defect)

Description

[Steffen Weber]

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.49 Safari/537.36

Steps to reproduce:

The HTML page firefox-test.html is served using SPDY. It contains 50 images (image-1.png ... image-50.png) that are served using SPDY, too. If these images are hosted on a different domain but on the same host/IP address as firefox-test.html then something goes wrong and Firefox loads these images very slowly.

I've tested this in Firefox 23 and "Aurora 25.0a2 (2013-08-09)" using a clean profile. The server software is nginx 1.4.2 (supports only spdy/2, no spdy/3). It works fine in Chrome 29 and Opera 12.

Notice that in each of the following 3 test-cases the file firefox-test.html has a different location but it always contains exactly the same HTML code that basically looks like this:

<li><img src="https://a1.betabase.de/image-1.png"></li>
...
<li><img src="https://a1.betabase.de/image-50.png"></li>

So no matter on which domain the file firefox-test.html is located, the images are always hosted at "a1.betabase.de".

The domains a1.betabase.de and a2.betabase.de are hosted on the same server but the domain b.betabase.de is hosted on a second server.

(I'm sorry for the invalid SSL certificates. I have verified that the bug also occurs with valid certificates (but cannot provide a testing environment on these production servers). If absolutely necessary I could buy valid certificates, please let me know.)

Test case 1: Images on same host and same domain as HTML page
1. Clear Browser Cache
2. Open https://a1.betabase.de/firefox-test.html
3. Notice that the images load quickly
=> Works :)

Test case 2: Images on same host but _different domain_ than HTML page
1. Clear Browser Cache
2. Open https://a2.betabase.de/firefox-test.html
3. Notice that the images load very slowly
=> Fails :(

Test case 3: Images on _different host_ and _different domain_ than HTML page
1. Clear Browser Cache
2. Open https://b.betabase.de/firefox-test.html
3. Notice that everything is fine again
=> Works :)

Please let me know if you need any additional information.

Comment 1

[Alice0775 White]

Confirmed in latest Nightly.
http://hg.mozilla.org/mozilla-central/rev/e33c2011643e
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0 ID:20130809030203

Comment 2

[Steffen Weber]

The problem disappears when I replace nginx 1.4.2 with Apache 2.2.25 + mod_spdy 0.9.3.3. So this might be a bug in nginx and not in Firefox.

See http://trac.nginx.org/nginx/ticket/403

Comment 3

[Patrick McManus [:mcmanus]]

Steffan - for test case 2 can you describe how slow is slow? I'm seeing the page load in less than 2 seconds, so I suspect I am not seeing what you are seeing. Can you still reproduce? Has the server config changed?

The interesting thing about case 2 is that if you had a real wildcard cert it would be eligible for spdy connection coalescing - we would send the requests for a1 and a2 on the same spdy connection because the cert proves they are authoritative for each other and they share the same IP. But we don't do that with certs bearing stored exceptions because we trust them less. But that's just a little tweak, not a massive deal.

If you're still having a problem you can get me some interesting information via an HTTP log:

https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging?redirectlocale=en-US&redirectslug=HTTP_Logging

fwiw - spdy/2 is soon to be retired in favor of adding spdy/3.1.. (in chrome too I believe) so you might be better off with mod_spdy for the moment.

Comment 4

[Steffen Weber]

Attachment: Firefox Bug 903484.mp4

Comment 5

[Steffen Weber]

Attachment: Firefox Bug 903484 Test-case 2.log.gz

Comment 6

[Steffen Weber]

Thank you for the reply. The server config has not changed.

I've added a few lines of JavaScript to the test case such that the page-load time is displayed in the document title. I've then created a screen capture (using Firefox 23 and ffmpeg): While test case 1 loads in less than 400 ms for me, test case 2 requires more than 4.000 ms to load. Please see the attached MP4 file.

As requested, I've also attached the HTTP log of running test-case 2. The page-load starts at 10:42:02.595380 and finishes about 4-5 seconds later.

I'm aware that spdy/2 is an old version of the protocol but nginx currently simply does not support anything newer.

Comment 7

[Patrick McManus [:mcmanus]]

your log shows you making a LOT of connections, while I see just the expected 2 - so that explains the different in observed results.

I believe this is an artifact of you not using a valid PKI wildcard cert to sign the domains.. (I'm not saying you should have to do that - I'm just saying I think that's what makes the issue come to light.)

My best theory is that firefox thinks it will be able to coalesce a1 onto a2 so it gracefully shuts down a1.. but when it actually tries to do that for a new image it realizes it can't (because the certs are too weak) and it has to make a new connection to a1.. and that cycle repeats again and again.

thanks for the log - it helps.

I suspect the reason I can't repro is that I'm farther away in terms of latency from your server.. so I manage to get more parallelism onto the wire before the graceful shutdown logic kicks in.

Steffen, I've made a theoretical fix that I would need you to verify. If it doesn't work the most useful thing to do is to send a new log from it. The builds will be available here in a few hours from this post:

http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-cfc02dfc2a98

Comment 8

[Steffen Weber]

Your try-build solves the problem! :)

Comment 9

[Steffen Weber]

If I understand your comment #7 correctly then we should be able to workaround this problem by using a wildcard certificate. Is this correct?

Comment 10

[Patrick McManus [:mcmanus]]

(In reply to Steffen Weber from comment #8)
> Your try-build solves the problem! :)

awesome. thanks for your help. I'll pretty up the patch for review.

(In reply to Steffen Weber from comment #9)
> If I understand your comment #7 correctly then we should be able to
> workaround this problem by using a wildcard certificate. Is this correct?

a wildcard cert from a CA that firefox recognizes. (i.e. not a self signed one)

Comment 11

[Patrick McManus [:mcmanus]]

Attachment: dont close spdy sessions with preferred entry if they can't actually coalesce yet

Comment 12

[Patrick McManus [:mcmanus]]

Steffen, thanks again for your help here. I needed to write a pretty different patch in the end because the last version reacted too strongly to ephemeral conditions that prevented the coalescing - especially TLS false start. If you get a chance and can verify a new build also addresses your problem, that would be a big help.

builds will be here (again in a couple of hours)

http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-39c4b5c3bf60

https://tbpl.mozilla.org/?tree=Try&rev=39c4b5c3bf60

Comment 13

[Steffen Weber]

The new try-build works, too! :)

(And, just to be sure, I've verified that the problem returns if I switch to Firefox 23.)

Comment 14

[u408661]

Comment on attachment 800351 [details] [diff] [review]
dont close spdy sessions with preferred entry if they can't actually coalesce yet

Review of attachment 800351 [details] [diff] [review]:
-----------------------------------------------------------------

One minor nit, LGTM otherwise.

::: netwerk/protocol/http/nsHttpConnectionMgr.cpp
@@ +570,5 @@
> +        // A connection entry (e.g. made with a different hostname) with
> +        // the same IP address is preferred for future transactions over this
> +        // connection entry. Gracefully close down the connection to help
> +        // new transactions migrate over.
> +        

nit: whitespace

Comment 15

[Patrick McManus [:mcmanus]]

remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/e08f432b9bab

Comment 16

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/e08f432b9bab

Comment 17

[Steffen Weber]

May I remove the test-cases or do you still need them?

Comment 18

[Patrick McManus [:mcmanus]]

(In reply to Steffen Weber from comment #17)
> May I remove the test-cases or do you still need them?

I don't need them. thanks.

Comment 19

[Steffen Weber]

Ok. And thank you very much for fixing this bug! :)