Closed Bug 1133598 Opened 5 years ago Closed 5 years ago

spdy/3.1 nginx malformed syn_reply

Categories

(Web Compatibility :: Desktop, defect)

Firefox 35
x86_64
Windows 7
defect
Not set

Tracking

(Not tracked)

RESOLVED INVALID

People

(Reporter: bugzilla.mozilla.org, Unassigned)

Details

(Whiteboard: [spdy])

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; InfoPath.3; rv:11.0) like Gecko

Steps to reproduce:

I have installed Owncloud on Nginx and enabled spdy.

**Operating system**:
Raspbian (Debian wheezy)
**Web server:**
Nginx 1.6.2
**PHP version:**
5.4.36-0+deb7u3 (php-fpm)
**ownCloud version:**
8.0 (stable) fresh install


Actual results:

I can't login to Owncloud with spdy suppport in Nginx.
I can login to Owncloud without spdy support in Nginx
I can login to Owncloud with spdy in Nginx and firefox about:config network.http.spdy.enabled.v3-1 set to false.

Tested with
Firefox 35.0.1 on Windows 7 64-bit
Firefox 38.0a1 (2015-02-16) on Windows 7 64-bit

Only when login fails Nginx debug logging shows:
2015/02/16 22:52:22 [info] 3691#0: *1 client sent WINDOW_UPDATE frame for unknown stream 273 while processing SPDY, client: 10.10.0.1, server: 0.0.0.0:443

I't shows this logging multiple times and only the 273 changes.


Expected results:

I should be able to login to Owncloud with a default Firefox configuration and spdy support on Nginx.
First posted remarks here:
https://bugzilla.mozilla.org/show_bug.cgi?id=992340
and was asked to make a report in Core/Networking.
Hi.. thanks for the report.

I doubt that nginx info log is at the heart of your problem. WINDOW_UPDATEs are basically akin to TCP acks and they often arrive after a sender has closed (i.e. made unknown) a stream - that race condition is inherent and expected.

from reading the linked bug it sounds like OPTIONS might be at the heart of the matter. That sounds much more likely to me - updates are a core part of the protocol and are very well exercised with nginx (e.g. via cloudflare web services) - but OPTIONS is a corner case that might well have something untested.

Do you have a server that can illustrate this for testing purposes? (Perhaps on a different port) .. you can mail me privately or make a private comment if you like.
Flags: needinfo?(bugzilla.mozilla.org)
Whiteboard: [spdy]
Just send you an email with access details of a 'server that can illustrate this for testing purposes'.
Flags: needinfo?(bugzilla.mozilla.org)
Thanks for sending the server. The problem is easily reproducible - and has nothing to do with either OPTIONS or WINDOW_UPDATE.

Firefox makes a POST for a login and the syn_reply (which contains the response headers) is consistently malformed exactly the same way each time. Firefox correctly generates a PROTOCOL_ERROR in response to the protocol error.

http://dev.chromium.org/spdy/spdy-protocol/spdy-protocol-draft3-1#TOC-2.6.2-SYN_REPLY

After decompressing the name/value header block, it is 1239 bytes long. I will put all 1239 uncompressed bytes into the next comment - these are sent by the server.

It indicates there are 16 header name value pairs. The first 14 are handled fine. After processing them there are 532 bytes remaining for the final 2 pairs. Each pair has a 8 byte overhead to encode the name and value lengths into 32 bit ints.

These are the remaining 532 bytes of the uncompressed name/value header block to start processing header 15

         00 00 00 0A 73 65 74 2D 63 6F 6F 6B 69 -< nvpair 15. "set-cookie" followed by 0xC1 (193) of value header
65 00 00 00 C1 6F 63 79 64 74 62 74 35 6D 70 6C
67 3D 36 61 6E 6C 6A 6C 36 6F 71 63 64 37 69 62
31 6E 66 70 63 38 61 35 72 71 36 32 3B 20 70 61
74 68 3D 2F 3B 20 73 65 63 75 72 65 3B 20 48 74
74 70 4F 6E 6C 79 00 6F 63 5F 75 73 65 72 6E 61
6D 65 3D 64 65 6C 65 74 65 64 3B 20 65 78 70 69
72 65 73 3D 54 68 75 2C 20 30 31 2D 4A 61 6E 2D
31 39 37 30 20 30 30 3A 30 30 3A 30 31 20 47 4D
54 00 6F 63 5F 74 6F 6B 65 6E 3D 64 65 6C 65 74
65 64 3B 20 65 78 70 69 72 65 73 3D 54 68 75 2C
20 30 31 2D 4A 61 6E 2D 31 39 37 30 20 30 30 3A
30 30 3A 30 31 20 47 4D 54 00 6F 63 5F 72 65 6D
65 6D 62 65 72 5F 6C 6F 67 69 6E 3D 64 65 6C 65 <- specified nvpair 16 should have namelen 6c 6f 67 69
74 65 64 3B 20 65 78 70 69 72 65 73 3D 54 68 75
2C 20 30 31 2D 4A 61 6E 2D 31 39 37 30 20 30 30
3A 30 30 3A 30 31 20 47 4D 54 00 6F 63 5F 75 73
65 72 6E 61 6D 65 3D 64 65 6C 65 74 65 64 3B 20
65 78 70 69 72 65 73 3D 54 68 75 2C 20 30 31 2D
4A 61 6E 2D 31 39 37 30 20 30 30 3A 30 30 3A 30
31 20 47 4D 54 3B 20 70 61 74 68 3D 2F 00 6F 63
5F 74 6F 6B 65 6E 3D 64 65 6C 65 74 65 64 3B 20
65 78 70 69 72 65 73 3D 54 68 75 2C 20 30 31 2D
4A 61 6E 2D 31 39 37 30 20 30 30 3A 30 30 3A 30
31 20 47 4D 54 3B 20 70 61 74 68 3D 2F 00 6F 63
5F 72 65 6D 65 6D 62 65 72 5F 6C 6F 67 69 6E 3D
64 65 6C 65 74 65 64 3B 20 65 78 70 69 72 65 73
3D 54 68 75 2C 20 30 31 2D 4A 61 6E 2D 31 39 37
30 20 30 30 3A 30 30 3A 30 31 20 47 4D 54 3B 20
70 61 74 68 3D 2F 00 00 00 08 6C 6F 63 61 74 69 <- probably real nvpair 16 namelen 00 00 00 08 "location"
6F 6E 00 00 00 31 68 74 74 70 73 3A 2F 2F 77 77
77 2E 70 6C 61 63 65 62 6F 72 75 6C 65 7A 2E 6E
6C 2F 69 6E 64 65 78 2E 70 68 70 2F 61 70 70 73
2F 66 69 6C 65 73 2F

the 00 00 00 0A means the name is 10 bytes. The next 10 bytes are "set-cookie".. so far so good.  Then 00 00 00 C1 means the next 193 bytes are the value.. in this case that is:

ocydtbt5mplg=l4up31jr99rj1g5ct5lpfr43h5; path=/; secure; HttpOnly[null]
oc_username=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT[null]
oc_token=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT[null]
oc_remember_

So that's 4 set-cookie h1 headers - each one separated by a null. The last one looks suspiciously truncated, but the spdy parser carries on looking for the last nvpair in the remaining 321 bytes. After the 193 bytes covered by the last value of header #15, that puts the length of header name #16 at the bytes "6c 6f 67 69" which is decimal 1819240297 - that's obviously way out of range and bigger than 321 - so firefox generates a PROTOCOL_ERROR and shuts down the session. That's why you can't login and firefox appears to be doing the right thing - this looks like an nginx bug.

If we look a little harder at that hex dump you can see if we interpret starting at the header name as acii it says

login=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT[null]
oc_username=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/[null]
oc_token=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/[null]
oc_remember_login=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/[null]

So in other words the length of the header value #14 was clearly encoded too short. Perhaps the expires or path information was automatically inserted and not counted in the length, I'm just speculating. Its an nginx issue.

If you skip across the bytes that look like those set-cookie headers (but are improperly encoded) you find the sequence "00 00 00 08" which looks more like the header name length for header #16. and indeed the next 8 bytes are "location", followed by "00 00 00 31" which would be a reasonable 49 bytes of header value - probably a url. (and there are 49 bytes left in the uncompressed block).
Summary: Bug with SPDY 3.1 - WINDOW_UPDATE frame for unknown stream → spdy/3.1 nginx malformed syn_reply
This is the full decompressed name/value header block that firefox generates PROTOCOL_ERROR on.

00 00 00 10 00 00 00 08 3A 76 65 72 73 69 6F 6E 
00 00 00 08 48 54 54 50 2F 31 2E 31 00 00 00 07 
3A 73 74 61 74 75 73 00 00 00 15 33 30 32 20 4D 
6F 76 65 64 20 54 65 6D 70 6F 72 61 72 69 6C 79 
00 00 00 06 73 65 72 76 65 72 00 00 00 05 6E 67 
69 6E 78 00 00 00 04 64 61 74 65 00 00 00 1D 54 
75 65 2C 20 31 37 20 46 65 62 20 32 30 31 35 20 
31 37 3A 34 33 3A 32 34 20 47 4D 54 00 00 00 0C 
63 6F 6E 74 65 6E 74 2D 74 79 70 65 00 00 00 18 
74 65 78 74 2F 68 74 6D 6C 3B 20 63 68 61 72 73 
65 74 3D 55 54 46 2D 38 00 00 00 07 65 78 70 69 
72 65 73 00 00 00 1D 54 68 75 2C 20 31 39 20 4E 
6F 76 20 31 39 38 31 20 30 38 3A 35 32 3A 30 30 
20 47 4D 54 00 00 00 0D 63 61 63 68 65 2D 63 6F 
6E 74 72 6F 6C 00 00 00 3E 6E 6F 2D 73 74 6F 72 
65 2C 20 6E 6F 2D 63 61 63 68 65 2C 20 6D 75 73 
74 2D 72 65 76 61 6C 69 64 61 74 65 2C 20 70 6F 
73 74 2D 63 68 65 63 6B 3D 30 2C 20 70 72 65 2D 
63 68 65 63 6B 3D 30 00 00 00 06 70 72 61 67 6D 
61 00 00 00 08 6E 6F 2D 63 61 63 68 65 00 00 00 
19 73 74 72 69 63 74 2D 74 72 61 6E 73 70 6F 72 
74 2D 73 65 63 75 72 69 74 79 00 00 00 23 6D 61 
78 2D 61 67 65 3D 33 31 35 33 36 30 30 30 3B 20 
69 6E 63 6C 75 64 65 53 75 62 44 6F 6D 61 69 6E 
73 00 00 00 10 78 2D 78 73 73 2D 70 72 6F 74 65 
63 74 69 6F 6E 00 00 00 0D 31 3B 20 6D 6F 64 65 
3D 62 6C 6F 63 6B 00 00 00 16 78 2D 63 6F 6E 74 
65 6E 74 2D 74 79 70 65 2D 6F 70 74 69 6F 6E 73 
00 00 00 07 6E 6F 73 6E 69 66 66 00 00 00 0F 78 
2D 66 72 61 6D 65 2D 6F 70 74 69 6F 6E 73 00 00 
00 0A 53 61 6D 65 6F 72 69 67 69 6E 00 00 00 17 
63 6F 6E 74 65 6E 74 2D 73 65 63 75 72 69 74 79 
2D 70 6F 6C 69 63 79 00 00 00 A0 64 65 66 61 75 
6C 74 2D 73 72 63 20 27 73 65 6C 66 27 3B 20 73 
63 72 69 70 74 2D 73 72 63 20 27 73 65 6C 66 27 
20 27 75 6E 73 61 66 65 2D 65 76 61 6C 27 3B 20 
73 74 79 6C 65 2D 73 72 63 20 27 73 65 6C 66 27 
20 27 75 6E 73 61 66 65 2D 69 6E 6C 69 6E 65 27 
3B 20 66 72 61 6D 65 2D 73 72 63 20 2A 3B 20 69 
6D 67 2D 73 72 63 20 2A 3B 20 66 6F 6E 74 2D 73 
72 63 20 27 73 65 6C 66 27 20 64 61 74 61 3A 3B 
20 6D 65 64 69 61 2D 73 72 63 20 2A 3B 20 63 6F 
6E 6E 65 63 74 2D 73 72 63 20 2A 00 00 00 0C 78 
2D 72 6F 62 6F 74 73 2D 74 61 67 00 00 00 04 6E 
6F 6E 65 00 00 00 0A 73 65 74 2D 63 6F 6F 6B 69 
65 00 00 00 C1 6F 63 79 64 74 62 74 35 6D 70 6C 
67 3D 36 74 75 65 71 71 70 30 6D 76 35 62 36 66 
6B 63 71 72 6B 75 73 65 6B 67 63 36 3B 20 70 61 
74 68 3D 2F 3B 20 73 65 63 75 72 65 3B 20 48 74 
74 70 4F 6E 6C 79 00 6F 63 5F 75 73 65 72 6E 61 
6D 65 3D 64 65 6C 65 74 65 64 3B 20 65 78 70 69 
72 65 73 3D 54 68 75 2C 20 30 31 2D 4A 61 6E 2D 
31 39 37 30 20 30 30 3A 30 30 3A 30 31 20 47 4D 
54 00 6F 63 5F 74 6F 6B 65 6E 3D 64 65 6C 65 74 
65 64 3B 20 65 78 70 69 72 65 73 3D 54 68 75 2C 
20 30 31 2D 4A 61 6E 2D 31 39 37 30 20 30 30 3A 
30 30 3A 30 31 20 47 4D 54 00 6F 63 5F 72 65 6D 
65 6D 62 65 72 5F 6C 6F 67 69 6E 3D 64 65 6C 65 
74 65 64 3B 20 65 78 70 69 72 65 73 3D 54 68 75 
2C 20 30 31 2D 4A 61 6E 2D 31 39 37 30 20 30 30 
3A 30 30 3A 30 31 20 47 4D 54 00 6F 63 5F 75 73 
65 72 6E 61 6D 65 3D 64 65 6C 65 74 65 64 3B 20 
65 78 70 69 72 65 73 3D 54 68 75 2C 20 30 31 2D 
4A 61 6E 2D 31 39 37 30 20 30 30 3A 30 30 3A 30 
31 20 47 4D 54 3B 20 70 61 74 68 3D 2F 00 6F 63 
5F 74 6F 6B 65 6E 3D 64 65 6C 65 74 65 64 3B 20 
65 78 70 69 72 65 73 3D 54 68 75 2C 20 30 31 2D 
4A 61 6E 2D 31 39 37 30 20 30 30 3A 30 30 3A 30 
31 20 47 4D 54 3B 20 70 61 74 68 3D 2F 00 6F 63 
5F 72 65 6D 65 6D 62 65 72 5F 6C 6F 67 69 6E 3D 
64 65 6C 65 74 65 64 3B 20 65 78 70 69 72 65 73 
3D 54 68 75 2C 20 30 31 2D 4A 61 6E 2D 31 39 37 
30 20 30 30 3A 30 30 3A 30 31 20 47 4D 54 3B 20 
70 61 74 68 3D 2F 00 00 00 08 6C 6F 63 61 74 69 
6F 6E 00 00 00 31 68 74 74 70 73 3A 2F 2F 77 77 
77 2E 70 6C 61 63 65 62 6F 72 75 6C 65 7A 2E 6E 
6C 2F 69 6E 64 65 78 2E 70 68 70 2F 61 70 70 73 
2F 66 69 6C 65 73 2F
Component: Networking → Networking: HTTP
Status: UNCONFIRMED → RESOLVED
Closed: 5 years ago
Resolution: --- → INVALID
I'll file a nginx bug and mark this as evangelism.
Status: RESOLVED → REOPENED
Component: Networking: HTTP → Desktop
Ever confirmed: true
Product: Core → Tech Evangelism
Resolution: INVALID → ---
Version: 38 Branch → Firefox 35
nginx replied to the bug and said it was a known and closed issue (starting with 1.7.2) where the size of header names and values were truncated at 8 bits, which is consistent with the data above.

so it sounds like the answer is upgrade nginx, or work around it with smaller cookies if you like to live dangerously :) (you need to consider the aggregate size of all the set-cookie headers, not just each line)

I'm going to mark this as invalid as there isn't a gecko bug here but I'm glad it reached a resolution.
Status: REOPENED → RESOLVED
Closed: 5 years ago5 years ago
Resolution: --- → INVALID
Thx for all your work on this Patrick!
I would not have guest a bug this 'big' to be in the latest release of Nginx stable.
But now reading up on Nginx releases:
http://nginx.com/blog/nginx-1-6-1-7-released/
and learing that it is 'better' to be on mainline than stable...
Just build and installed Nginx 1.7.10 and it's confirmed fixed!
Thx again for your help!
and I updated nginx on one of my boxes using this development ubuntu ppa - https://launchpad.net/~nginx/+archive/ubuntu/development

might be handy to have it in a comment here for someone else.
Product: Tech Evangelism → Web Compatibility
You need to log in before you can comment on or make changes to this bug.