629427 - "Disallowed key characters in global data." on crash-stats

Status: RESOLVED FIXED

(Socorro :: Webapp, task)

Description

[Robert Helmer [:rhelmer]]

jst reported a problem in #socorro around 9:03 this morning:

17:03 < jst> so I was trying to load 
             http://crash-stats.mozilla.com/products/Firefox
17:04 < jst> and I get a page that says "Disallowed key characters in global 
             data."

lars suggested clearing cookies, which seems to have fixed for jst. lars could also reproduce this (http only, not with https) and clearing cookies seems to have helped.

This is not an expected error (we only saw it in SJC when it was in a really bad state, postgres was down. There is an "orange screen" kohana error related to cookies when going from 1.7.6 to an earlier version, but we don't think this is it).

stephend reproduced this at 9:07, with user agent 3.6.13 on NT 6.1. I haven't seen an error around this time, going to try to track it down in the access logs.

I haven't found anything untoward in the logs.

We want to:

1) make sure this is not a result of traffic going to SJC (it should not be at this point, the old loadbalancer is sending all traffic to PHX)
2) understand why the error is happening in PHX, if that's the case

One thing ryansnyder noticed is that the error happened on a http URL, but we should be redirecting those consistently to https. May or may not be related, there will be a separate bug for that.

Comment 1

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

(In reply to comment #0)
> lars suggested clearing cookies, which seems to have fixed for jst. lars could
> also reproduce this (http only, not with https) and clearing cookies seems to
> have helped.

I saw this last night, too (I forget at what time, and I forgot to look for debug output on my end) trying to load http://crash-stats.mozilla.com/ in a completely fresh profile (so no cookies at all) when I was testing a patch.  Switching to https loaded correctly.

Comment 2

[Robert Helmer [:rhelmer]]

Kohana tries to match a regex to any GET, POST or cookie which causes this message:

http://code.google.com/p/socorro/source/browse/tags/releases/1.7.6_r2892_20110121/webapp-php/system/libraries/Input.php#391

As far as I know this has only been reproduced on http not https, it is possible that this was an existing bug that we have not seen because we were consistently redirecting to https before.

Comment 3

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

I have a hunch that switching to https will fix the issue.  We have a few
cookies that are https only, which made me wonder if that was a contributing
or root cause of the Disallowed keys error.

Comment 4

[Laura Thomson :laura]

If we were consistently redirecting to https before, let's do that here as well.  Let's do it anyway, I'm pretty sure the infra can cope with the extra load.

Comment 5

[Stephen Donner [:stephend] Not actively reading bugmail]

David, we need to change baseurl to point to https:// (http://viewvc.svn.mozilla.org/vc/projects/socorro_qa/vars.py?revision=81228&view=markup).  rhelmer also suggests we "put a check in there to make sure http:// gets redirected to https://" after the initial GET request.

Comment 6

[Robert Helmer [:rhelmer]]

Jabba, can we compare PHX web configs against the SJC configs? An easy way to do this is to drop the SJC ones into prod/etc-socorro/web/ and run "svn diff".

Maybe we should find a place in puppet's SVN to check in the old configs, just for archival purposes?

I took a look at the current configs and they seem ok, but it's possible we're missing something we had in SJC. The web app should be doing the redirect, so it should not be dependent on Apache or php.ini settings as far as I can tell.

Comment 7

[Laura Thomson :laura]

I'm reproing this bug again now.

Comment 8

[Justin Dow [:jabba]]

This happened over the weekend randomly and then cleared again. Note, that I've already updated the configs to force https, so that didn't seem to fix it.

Comment 9

[Robert Helmer [:rhelmer]]

(In reply to comment #2)
> Kohana tries to match a regex to any GET, POST or cookie which causes this
> message:
> 
> http://code.google.com/p/socorro/source/browse/tags/releases/1.7.6_r2892_20110121/webapp-php/system/libraries/Input.php#391
> 
> As far as I know this has only been reproduced on http not https, it is
> possible that this was an existing bug that we have not seen because we were
> consistently redirecting to https before.

Does not seem to have anything to do with HTTP versus HTTPS, and probably not caching related since nagios noticed the problem on each individual web server.

I think the next step should be to add some debug logging to the clean_input_keys() function, and see exactly what it's failing on. 

The nagios check does not use cookies AFAIK and does the same GET over and over, so I can't imagine what is triggering this regex.

Comment 10

[Justin Dow [:jabba]]

FYI: we didn't actually prove that zeus cache isn't the problem. the http checks on the webheads hit the webhead, get redirected to https://crash-stats.mozilla.com/ and therefore hit the zeus, which is why all 5 of them paged at the same time...

Comment 11

[Justin Dow [:jabba]]

I turned off caching in zeus just now.

Comment 12

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to comment #11)
> I turned off caching in zeus just now.

Our Socorro tests have consistently been passing, since: http://qa-selenium.mv.mozilla.com:8080/job/socorro/.  I'll keep an eye on them.

Comment 13

[Laura Thomson :laura]

Fixed by Django