Closed Bug 1133598 Opened 10 years ago Closed 10 years ago

spdy/3.1 nginx malformed syn_reply

Categories

(Web Compatibility :: Site Reports, defect)

Product:
Web Compatibility
Component:
Site Reports
Version:
Firefox 35
Platform:
x86_64
Windows 7
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED INVALID

People

(Reporter: bugzilla.mozilla.org, Unassigned)

Details

(Whiteboard: [spdy])

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; InfoPath.3; rv:11.0) like Gecko Steps to reproduce: I have installed Owncloud on Nginx and enabled spdy. **Operating system**: Raspbian (Debian wheezy) **Web server:** Nginx 1.6.2 **PHP version:** 5.4.36-0+deb7u3 (php-fpm) **ownCloud version:** 8.0 (stable) fresh install Actual results: I can't login to Owncloud with spdy suppport in Nginx. I can login to Owncloud without spdy support in Nginx I can login to Owncloud with spdy in Nginx and firefox about:config network.http.spdy.enabled.v3-1 set to false. Tested with Firefox 35.0.1 on Windows 7 64-bit Firefox 38.0a1 (2015-02-16) on Windows 7 64-bit Only when login fails Nginx debug logging shows: 2015/02/16 22:52:22 [info] 3691#0: *1 client sent WINDOW_UPDATE frame for unknown stream 273 while processing SPDY, client: 10.10.0.1, server: 0.0.0.0:443 I't shows this logging multiple times and only the 273 changes. Expected results: I should be able to login to Owncloud with a default Firefox configuration and spdy support on Nginx.
First posted remarks here: https://bugzilla.mozilla.org/show_bug.cgi?id=992340 and was asked to make a report in Core/Networking.
Hi.. thanks for the report. I doubt that nginx info log is at the heart of your problem. WINDOW_UPDATEs are basically akin to TCP acks and they often arrive after a sender has closed (i.e. made unknown) a stream - that race condition is inherent and expected. from reading the linked bug it sounds like OPTIONS might be at the heart of the matter. That sounds much more likely to me - updates are a core part of the protocol and are very well exercised with nginx (e.g. via cloudflare web services) - but OPTIONS is a corner case that might well have something untested. Do you have a server that can illustrate this for testing purposes? (Perhaps on a different port) .. you can mail me privately or make a private comment if you like.
Flags: needinfo?(bugzilla.mozilla.org)
Whiteboard: [spdy]
Just send you an email with access details of a 'server that can illustrate this for testing purposes'.
Flags: needinfo?(bugzilla.mozilla.org)
Thanks for sending the server. The problem is easily reproducible - and has nothing to do with either OPTIONS or WINDOW_UPDATE. Firefox makes a POST for a login and the syn_reply (which contains the response headers) is consistently malformed exactly the same way each time. Firefox correctly generates a PROTOCOL_ERROR in response to the protocol error. http://dev.chromium.org/spdy/spdy-protocol/spdy-protocol-draft3-1#TOC-2.6.2-SYN_REPLY After decompressing the name/value header block, it is 1239 bytes long. I will put all 1239 uncompressed bytes into the next comment - these are sent by the server. It indicates there are 16 header name value pairs. The first 14 are handled fine. After processing them there are 532 bytes remaining for the final 2 pairs. Each pair has a 8 byte overhead to encode the name and value lengths into 32 bit ints. These are the remaining 532 bytes of the uncompressed name/value header block to start processing header 15 00 00 00 0A 73 65 74 2D 63 6F 6F 6B 69 -< nvpair 15. "set-cookie" followed by 0xC1 (193) of value header 65 00 00 00 C1 6F 63 79 64 74 62 74 35 6D 70 6C 67 3D 36 61 6E 6C 6A 6C 36 6F 71 63 64 37 69 62 31 6E 66 70 63 38 61 35 72 71 36 32 3B 20 70 61 74 68 3D 2F 3B 20 73 65 63 75 72 65 3B 20 48 74 74 70 4F 6E 6C 79 00 6F 63 5F 75 73 65 72 6E 61 6D 65 3D 64 65 6C 65 74 65 64 3B 20 65 78 70 69 72 65 73 3D 54 68 75 2C 20 30 31 2D 4A 61 6E 2D 31 39 37 30 20 30 30 3A 30 30 3A 30 31 20 47 4D 54 00 6F 63 5F 74 6F 6B 65 6E 3D 64 65 6C 65 74 65 64 3B 20 65 78 70 69 72 65 73 3D 54 68 75 2C 20 30 31 2D 4A 61 6E 2D 31 39 37 30 20 30 30 3A 30 30 3A 30 31 20 47 4D 54 00 6F 63 5F 72 65 6D 65 6D 62 65 72 5F 6C 6F 67 69 6E 3D 64 65 6C 65 <- specified nvpair 16 should have namelen 6c 6f 67 69 74 65 64 3B 20 65 78 70 69 72 65 73 3D 54 68 75 2C 20 30 31 2D 4A 61 6E 2D 31 39 37 30 20 30 30 3A 30 30 3A 30 31 20 47 4D 54 00 6F 63 5F 75 73 65 72 6E 61 6D 65 3D 64 65 6C 65 74 65 64 3B 20 65 78 70 69 72 65 73 3D 54 68 75 2C 20 30 31 2D 4A 61 6E 2D 31 39 37 30 20 30 30 3A 30 30 3A 30 31 20 47 4D 54 3B 20 70 61 74 68 3D 2F 00 6F 63 5F 74 6F 6B 65 6E 3D 64 65 6C 65 74 65 64 3B 20 65 78 70 69 72 65 73 3D 54 68 75 2C 20 30 31 2D 4A 61 6E 2D 31 39 37 30 20 30 30 3A 30 30 3A 30 31 20 47 4D 54 3B 20 70 61 74 68 3D 2F 00 6F 63 5F 72 65 6D 65 6D 62 65 72 5F 6C 6F 67 69 6E 3D 64 65 6C 65 74 65 64 3B 20 65 78 70 69 72 65 73 3D 54 68 75 2C 20 30 31 2D 4A 61 6E 2D 31 39 37 30 20 30 30 3A 30 30 3A 30 31 20 47 4D 54 3B 20 70 61 74 68 3D 2F 00 00 00 08 6C 6F 63 61 74 69 <- probably real nvpair 16 namelen 00 00 00 08 "location" 6F 6E 00 00 00 31 68 74 74 70 73 3A 2F 2F 77 77 77 2E 70 6C 61 63 65 62 6F 72 75 6C 65 7A 2E 6E 6C 2F 69 6E 64 65 78 2E 70 68 70 2F 61 70 70 73 2F 66 69 6C 65 73 2F the 00 00 00 0A means the name is 10 bytes. The next 10 bytes are "set-cookie".. so far so good. Then 00 00 00 C1 means the next 193 bytes are the value.. in this case that is: ocydtbt5mplg=l4up31jr99rj1g5ct5lpfr43h5; path=/; secure; HttpOnly[null] oc_username=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT[null] oc_token=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT[null] oc_remember_ So that's 4 set-cookie h1 headers - each one separated by a null. The last one looks suspiciously truncated, but the spdy parser carries on looking for the last nvpair in the remaining 321 bytes. After the 193 bytes covered by the last value of header #15, that puts the length of header name #16 at the bytes "6c 6f 67 69" which is decimal 1819240297 - that's obviously way out of range and bigger than 321 - so firefox generates a PROTOCOL_ERROR and shuts down the session. That's why you can't login and firefox appears to be doing the right thing - this looks like an nginx bug. If we look a little harder at that hex dump you can see if we interpret starting at the header name as acii it says login=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT[null] oc_username=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/[null] oc_token=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/[null] oc_remember_login=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/[null] So in other words the length of the header value #14 was clearly encoded too short. Perhaps the expires or path information was automatically inserted and not counted in the length, I'm just speculating. Its an nginx issue. If you skip across the bytes that look like those set-cookie headers (but are improperly encoded) you find the sequence "00 00 00 08" which looks more like the header name length for header #16. and indeed the next 8 bytes are "location", followed by "00 00 00 31" which would be a reasonable 49 bytes of header value - probably a url. (and there are 49 bytes left in the uncompressed block).
Summary: Bug with SPDY 3.1 - WINDOW_UPDATE frame for unknown stream → spdy/3.1 nginx malformed syn_reply
This is the full decompressed name/value header block that firefox generates PROTOCOL_ERROR on. 00 00 00 10 00 00 00 08 3A 76 65 72 73 69 6F 6E 00 00 00 08 48 54 54 50 2F 31 2E 31 00 00 00 07 3A 73 74 61 74 75 73 00 00 00 15 33 30 32 20 4D 6F 76 65 64 20 54 65 6D 70 6F 72 61 72 69 6C 79 00 00 00 06 73 65 72 76 65 72 00 00 00 05 6E 67 69 6E 78 00 00 00 04 64 61 74 65 00 00 00 1D 54 75 65 2C 20 31 37 20 46 65 62 20 32 30 31 35 20 31 37 3A 34 33 3A 32 34 20 47 4D 54 00 00 00 0C 63 6F 6E 74 65 6E 74 2D 74 79 70 65 00 00 00 18 74 65 78 74 2F 68 74 6D 6C 3B 20 63 68 61 72 73 65 74 3D 55 54 46 2D 38 00 00 00 07 65 78 70 69 72 65 73 00 00 00 1D 54 68 75 2C 20 31 39 20 4E 6F 76 20 31 39 38 31 20 30 38 3A 35 32 3A 30 30 20 47 4D 54 00 00 00 0D 63 61 63 68 65 2D 63 6F 6E 74 72 6F 6C 00 00 00 3E 6E 6F 2D 73 74 6F 72 65 2C 20 6E 6F 2D 63 61 63 68 65 2C 20 6D 75 73 74 2D 72 65 76 61 6C 69 64 61 74 65 2C 20 70 6F 73 74 2D 63 68 65 63 6B 3D 30 2C 20 70 72 65 2D 63 68 65 63 6B 3D 30 00 00 00 06 70 72 61 67 6D 61 00 00 00 08 6E 6F 2D 63 61 63 68 65 00 00 00 19 73 74 72 69 63 74 2D 74 72 61 6E 73 70 6F 72 74 2D 73 65 63 75 72 69 74 79 00 00 00 23 6D 61 78 2D 61 67 65 3D 33 31 35 33 36 30 30 30 3B 20 69 6E 63 6C 75 64 65 53 75 62 44 6F 6D 61 69 6E 73 00 00 00 10 78 2D 78 73 73 2D 70 72 6F 74 65 63 74 69 6F 6E 00 00 00 0D 31 3B 20 6D 6F 64 65 3D 62 6C 6F 63 6B 00 00 00 16 78 2D 63 6F 6E 74 65 6E 74 2D 74 79 70 65 2D 6F 70 74 69 6F 6E 73 00 00 00 07 6E 6F 73 6E 69 66 66 00 00 00 0F 78 2D 66 72 61 6D 65 2D 6F 70 74 69 6F 6E 73 00 00 00 0A 53 61 6D 65 6F 72 69 67 69 6E 00 00 00 17 63 6F 6E 74 65 6E 74 2D 73 65 63 75 72 69 74 79 2D 70 6F 6C 69 63 79 00 00 00 A0 64 65 66 61 75 6C 74 2D 73 72 63 20 27 73 65 6C 66 27 3B 20 73 63 72 69 70 74 2D 73 72 63 20 27 73 65 6C 66 27 20 27 75 6E 73 61 66 65 2D 65 76 61 6C 27 3B 20 73 74 79 6C 65 2D 73 72 63 20 27 73 65 6C 66 27 20 27 75 6E 73 61 66 65 2D 69 6E 6C 69 6E 65 27 3B 20 66 72 61 6D 65 2D 73 72 63 20 2A 3B 20 69 6D 67 2D 73 72 63 20 2A 3B 20 66 6F 6E 74 2D 73 72 63 20 27 73 65 6C 66 27 20 64 61 74 61 3A 3B 20 6D 65 64 69 61 2D 73 72 63 20 2A 3B 20 63 6F 6E 6E 65 63 74 2D 73 72 63 20 2A 00 00 00 0C 78 2D 72 6F 62 6F 74 73 2D 74 61 67 00 00 00 04 6E 6F 6E 65 00 00 00 0A 73 65 74 2D 63 6F 6F 6B 69 65 00 00 00 C1 6F 63 79 64 74 62 74 35 6D 70 6C 67 3D 36 74 75 65 71 71 70 30 6D 76 35 62 36 66 6B 63 71 72 6B 75 73 65 6B 67 63 36 3B 20 70 61 74 68 3D 2F 3B 20 73 65 63 75 72 65 3B 20 48 74 74 70 4F 6E 6C 79 00 6F 63 5F 75 73 65 72 6E 61 6D 65 3D 64 65 6C 65 74 65 64 3B 20 65 78 70 69 72 65 73 3D 54 68 75 2C 20 30 31 2D 4A 61 6E 2D 31 39 37 30 20 30 30 3A 30 30 3A 30 31 20 47 4D 54 00 6F 63 5F 74 6F 6B 65 6E 3D 64 65 6C 65 74 65 64 3B 20 65 78 70 69 72 65 73 3D 54 68 75 2C 20 30 31 2D 4A 61 6E 2D 31 39 37 30 20 30 30 3A 30 30 3A 30 31 20 47 4D 54 00 6F 63 5F 72 65 6D 65 6D 62 65 72 5F 6C 6F 67 69 6E 3D 64 65 6C 65 74 65 64 3B 20 65 78 70 69 72 65 73 3D 54 68 75 2C 20 30 31 2D 4A 61 6E 2D 31 39 37 30 20 30 30 3A 30 30 3A 30 31 20 47 4D 54 00 6F 63 5F 75 73 65 72 6E 61 6D 65 3D 64 65 6C 65 74 65 64 3B 20 65 78 70 69 72 65 73 3D 54 68 75 2C 20 30 31 2D 4A 61 6E 2D 31 39 37 30 20 30 30 3A 30 30 3A 30 31 20 47 4D 54 3B 20 70 61 74 68 3D 2F 00 6F 63 5F 74 6F 6B 65 6E 3D 64 65 6C 65 74 65 64 3B 20 65 78 70 69 72 65 73 3D 54 68 75 2C 20 30 31 2D 4A 61 6E 2D 31 39 37 30 20 30 30 3A 30 30 3A 30 31 20 47 4D 54 3B 20 70 61 74 68 3D 2F 00 6F 63 5F 72 65 6D 65 6D 62 65 72 5F 6C 6F 67 69 6E 3D 64 65 6C 65 74 65 64 3B 20 65 78 70 69 72 65 73 3D 54 68 75 2C 20 30 31 2D 4A 61 6E 2D 31 39 37 30 20 30 30 3A 30 30 3A 30 31 20 47 4D 54 3B 20 70 61 74 68 3D 2F 00 00 00 08 6C 6F 63 61 74 69 6F 6E 00 00 00 31 68 74 74 70 73 3A 2F 2F 77 77 77 2E 70 6C 61 63 65 62 6F 72 75 6C 65 7A 2E 6E 6C 2F 69 6E 64 65 78 2E 70 68 70 2F 61 70 70 73 2F 66 69 6C 65 73 2F
Component: Networking → Networking: HTTP
Status: UNCONFIRMED → RESOLVED
Closed: 10 years ago
Resolution: --- → INVALID
I'll file a nginx bug and mark this as evangelism.
Status: RESOLVED → REOPENED
Component: Networking: HTTP → Desktop
Ever confirmed: true
Product: Core → Tech Evangelism
Resolution: INVALID → ---
Version: 38 Branch → Firefox 35
nginx replied to the bug and said it was a known and closed issue (starting with 1.7.2) where the size of header names and values were truncated at 8 bits, which is consistent with the data above. so it sounds like the answer is upgrade nginx, or work around it with smaller cookies if you like to live dangerously :) (you need to consider the aggregate size of all the set-cookie headers, not just each line) I'm going to mark this as invalid as there isn't a gecko bug here but I'm glad it reached a resolution.
Status: REOPENED → RESOLVED
Closed: 10 years ago → 10 years ago
Resolution: --- → INVALID
Thx for all your work on this Patrick! I would not have guest a bug this 'big' to be in the latest release of Nginx stable. But now reading up on Nginx releases: http://nginx.com/blog/nginx-1-6-1-7-released/ and learing that it is 'better' to be on mainline than stable...
Just build and installed Nginx 1.7.10 and it's confirmed fixed! Thx again for your help!
and I updated nginx on one of my boxes using this development ubuntu ppa - https://launchpad.net/~nginx/+archive/ubuntu/development might be handy to have it in a comment here for someone else.
Product: Tech Evangelism → Web Compatibility
You need to log in before you can comment on or make changes to this bug.