516765 - Disallowed key characters in global data on homepage

Status: RESOLVED INCOMPLETE

(Websites Graveyard :: mozillaservice.org, defect)

Description

[Austin King [:ozten]]

The homepage is occasionally giving
"Disallowed key characters in global data."
instead of the expected content.

Example of Output:
http://mozillaservice.org/?id%27=a

This is reported to have happened roughly at
9/14 4-5pm PDT (release time)

9/15 2:30am PDT
9/15 9am PDT

This was also in the google search cache for a atleast 30 minutes yesterday, but is back to normal content.

Comment 1

[Austin King [:ozten]]

The codepath which causes this is related to the Input library which is used to protect against XSS attacks by inspecting GET, POST, and Cookie inputs.

Example: Loading the homepage from another page will cause
 the following cookie keys to be exampled:
__unam
lang
kohanasession
authautologin
s_sq
s_cc
kohanasession_data

In our bad output example above, we'll die on inspecting
id'
since ' is not valid in a key name.

Comment 2

[Austin King [:ozten]]

Checking in a patch to get it onto stage for testing sooner than later. Still waiting for prod access logs to troubleshoot.

Comment 3

[Michael Morgan [:morgamic]]

Suggestions:
1) remove the exit() and instead reply with something better
2) just wipe the inputs and redirect to homepage
3) add no-cache headers prior to exit() callback (http://morgamic.pastebin.mozilla.org/671271) to ensure the error (when it happens) never gets cached by the proxy cache

Comment 4

[Michael Morgan [:morgamic]]

Austin - just posted that in case it helps, those were my notes from yesterday.

Comment 5

[Austin King [:ozten]]

(In reply to comment #3)
On error, I included your patch as well as

* Sending 503 Service Unavailable
* Logging the HTTP_SERVER_VARS
* Logging the _REQUEST

The bad key has to be either in the headers, query string, or body of the request. This will help us identify which.

Comment 6

[Austin King [:ozten]]

Here are all the requests and user agents from yesterday and today where I can detect them getting this error:

/?from=sfx&uid=0&t=475 HTTP/1.1"Python
/?from=sfx&uid=0&t=476 HTTP/1.1"Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent
?from=sfx&uid=0&t=476/ HTTP/1.1"Python
/?from=sfx&uid=0&t=477 HTTP/1.1"Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent
?from=sfx&uid=104854&t=476/ HTTP/1.1"Python
/?from=sfx&uid=259715&t=477 HTTP/1.1"Mozilla/5.0 (en
/?from=sfx&uid=287227&t=475 HTTP/1.1"Jakarta Commons
/?from=sfx&uid=287902&t=476 HTTP/1.1"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
/?from=sfx&uid=91977&t=475 HTTP/1.0"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) Fetch API Request
/?from=sfx&uid=91977&t=476 HTTP/1.1"Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent
/home/index/es_ES?from=sfx&uid=0&t=500 HTTP/1.1"Mozilla/5.0 (Windows; U; Windows NT 5.2; en
/home/index/pt_BR?from=sfx&uid=0&t=530 HTTP/1.1"Mozilla/5.0 (en
http://mozillaservice.org/learn_more/index/en_US/?wtf=!%29!@%28*@&%29%28@*"
/?id%27=a HTTP/1.1"Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en
/?id=%29*@~!_@%29%28~_%29$*!%29*&% HTTP/1.1"Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en
/?id=)*@~!_@)(~_)$*!)*&% HTTP/1.1"Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en
/learn_more/index/en_US/?wtf=!%29!@%28*@&%29%28@* HTTP/1.1"Mozilla/5.0 (X11; U; Linux x86_64; en
/learn_more/index/en_US/?wtf=!)!@(*@&)(@* HTTP/1.1"Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en
/?wtf=81098213901872%28*&@%29*%28&!&@@ HTTP/1.1"Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en
/?wtf=81098213901872%28*&@%29*%28&!&@@ HTTP/1.1"Mozilla/5.0 (X11; U; Linux x86_64; en
/?wtf=81098213901872(*&@)*(&!&@@ HTTP/1.1"Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en

Comment 7

[Austin King [:ozten]]

/?from=sfx - these are our links (all over the web) via Spreadfire Badges. Probably a poorly written spider isn't unencoing the url properly

/?id%27 and ?wtf - these are morgamic messing with us :)

Comment 8

[Austin King [:ozten]]

Under normal circumstances the homepage isn't cached.
I've verified this with curl:
http://pastebin.mozilla.org/671419

If that's true, then a spider should be the root cause of the issue.

Comment 9

[Austin King [:ozten]]

(In reply to comment #8)
Typo: If that's true, then a spider should *not* be the root cause of the issue.


Please update this bug if you see the issue again. 

We have deployed some instrumentation to production and can pull a log for that date/time to troubleshoot this issue.