1043402 - SPDY/H2 considers compression decoding errors as session errors

Status: RESOLVED FIXED

(Core :: Networking: HTTP, defect)

Description

[James Willcox (:snorp) ([e-mail]) (he/him)]

It appears that Nightly has some problems with CORS requests going over a SPDY proxy. Using Grafana (which does a ton of XHR), some requests randomly fail. This is going through janus.allizom.org. Of course the proxy could also be at fault here, but it worked through Chrome, so I'm pointing the finger at Gecko for now :)

The actual page in question is protected, but I can give access to whoever needs it.

Comment 1

[Patrick McManus [:mcmanus]]

I've found a couple of interesting things

1] I'm pretty certain janus is sometimes returning documents (they seem to always be css for me, but its not always the same css) that are marked content-encoding: gzip but are not actually gzipped. This results in a stream error and broken css. I'll attach an example below.

2] There is a spdy bug that considers that problem a session level error instead of a stream level error. So we close down all in progress streams - not just the one with the bogus content-encoding. This makes it impact far more severely that it should - which could explain why it is more visible in firefox than chrome. I'll fix this part.

I haven't seen a cors problem yet, but 

http request [
  GET http://people.mozilla.org/~esawin/grafana/css/default.min.a9bd22e8.css HTTP/1.1
  Host: people.mozilla.org
  User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:34.0) Gecko/20100101 Firefox/34.0
  Accept: text/css,*/*;q=0.1
  Accept-Language: en-US,en;q=0.5
  Accept-Encoding: gzip, deflate
  Referer: http://people.mozilla.org/~esawin/grafana/
  Cookie: optimizelySegments=%7B%22245875585%22%3A%22direct%22%2C%22245617832%22%3A%22none%22%2C%22246048108%22%3A%22false%22%2C%22245677587%22%3A%22ff%22%2C%22869421433%22%3A%22true%22%7D; optimizelyEndUserId=oeu1406746291521r0.45404266944308613; optimizelyBuck\
ets=%7B%7D; __utma=150903082.1137603889.1406746293.1406746293.1406746293.1; __utmz=150903082.1406746293.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)
  Connection: keep-alive
]

http response [
   HTTP/1.1 200 OK
   Date: Fri, 01 Aug 2014 13:41:55 GMT
   Server: Apache
   x-backend-server: people1.dmz.scl3.mozilla.com
   Last-Modified: Tue, 24 Jun 2014 13:07:56 GMT
   Etag: "31ef4"
   Accept-Ranges: bytes
   Vary: Accept-Encoding
   Content-Type: text/css
   via: janus 0.2.1
   x-original-content-length: 204532
   Content-Encoding: gzip
   X-Firefox-Spdy: 3.1
]

(just the first few bytes shown)
id=0x3 [Reading Data Frame]
40 63 68 61 72 73 65 74 20 22 55 54 46 2D 38 22
3B 2F 2A 21 20 6E 6F 72 6D 61 6C 69 7A 65 2E 63
73 73 20 76 31 2E 30 2E 31 20 7C 20 4D 49 54 20

note that 40 63 68 61 is "@cha" which are the plaintext leading bytes
of that css.. so no gzip.

Comment 2

[Patrick McManus [:mcmanus]]

as for the CORS thing, I don't see anything CORS specific - but I do see some fraction of the XHRs in dev tools as having 0 bytes (because we aborted on the decoding error) - and when I drill down on them they all are gzip encoded json. I believe they are having the same problem.

(its also quite possible that chrome is auto detecting identity even when you say gzip so its just tolerant of that server error).

Comment 3

[Patrick McManus [:mcmanus]]

Your results will be at https://tbpl.mozilla.org/?tree=Try&rev=7675579ae672

Comment 4

[Patrick McManus [:mcmanus]]

Attachment: spdy/h2 ce errors should be stream local

Comment 5

[u408661]

Comment on attachment 8466205 [details] [diff] [review]
spdy/h2 ce errors should be stream local

Review of attachment 8466205 [details] [diff] [review]:
-----------------------------------------------------------------

Ah, the glorious, ever-expanding SoftStreamError :)

Comment 6

[James Willcox (:snorp) ([e-mail]) (he/him)]

I can't seem to reproduce this gzip issue or the original CORS one anymore on Nightly, so that's interesting.

Comment 7

[Patrick McManus [:mcmanus]]

  https://hg.mozilla.org/integration/mozilla-inbound/rev/ce89d709b63c

Comment 8

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/ce89d709b63c