"Disallowed key characters in global data." on crash-stats

RESOLVED FIXED

Status

Socorro
Webapp
RESOLVED FIXED
7 years ago
5 years ago

People

(Reporter: rhelmer, Unassigned)

Tracking

Bug Flags:
in-testsuite ?

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

7 years ago
jst reported a problem in #socorro around 9:03 this morning:

17:03 < jst> so I was trying to load 
             http://crash-stats.mozilla.com/products/Firefox
17:04 < jst> and I get a page that says "Disallowed key characters in global 
             data."

lars suggested clearing cookies, which seems to have fixed for jst. lars could also reproduce this (http only, not with https) and clearing cookies seems to have helped.

This is not an expected error (we only saw it in SJC when it was in a really bad state, postgres was down. There is an "orange screen" kohana error related to cookies when going from 1.7.6 to an earlier version, but we don't think this is it).

stephend reproduced this at 9:07, with user agent 3.6.13 on NT 6.1. I haven't seen an error around this time, going to try to track it down in the access logs.

I haven't found anything untoward in the logs.

We want to:

1) make sure this is not a result of traffic going to SJC (it should not be at this point, the old loadbalancer is sending all traffic to PHX)
2) understand why the error is happening in PHX, if that's the case

One thing ryansnyder noticed is that the error happened on a http URL, but we should be redirecting those consistently to https. May or may not be related, there will be a separate bug for that.
(In reply to comment #0)
> lars suggested clearing cookies, which seems to have fixed for jst. lars could
> also reproduce this (http only, not with https) and clearing cookies seems to
> have helped.

I saw this last night, too (I forget at what time, and I forgot to look for debug output on my end) trying to load http://crash-stats.mozilla.com/ in a completely fresh profile (so no cookies at all) when I was testing a patch.  Switching to https loaded correctly.
(Reporter)

Comment 2

7 years ago
Kohana tries to match a regex to any GET, POST or cookie which causes this message:

http://code.google.com/p/socorro/source/browse/tags/releases/1.7.6_r2892_20110121/webapp-php/system/libraries/Input.php#391

As far as I know this has only been reproduced on http not https, it is possible that this was an existing bug that we have not seen because we were consistently redirecting to https before.
I have a hunch that switching to https will fix the issue.  We have a few
cookies that are https only, which made me wonder if that was a contributing
or root cause of the Disallowed keys error.

Comment 4

7 years ago
If we were consistently redirecting to https before, let's do that here as well.  Let's do it anyway, I'm pretty sure the infra can cope with the extra load.
David, we need to change baseurl to point to https:// (http://viewvc.svn.mozilla.org/vc/projects/socorro_qa/vars.py?revision=81228&view=markup).  rhelmer also suggests we "put a check in there to make sure http:// gets redirected to https://" after the initial GET request.
Flags: in-testsuite?
OS: Linux → All
Hardware: x86_64 → All
(Reporter)

Comment 6

7 years ago
Jabba, can we compare PHX web configs against the SJC configs? An easy way to do this is to drop the SJC ones into prod/etc-socorro/web/ and run "svn diff".

Maybe we should find a place in puppet's SVN to check in the old configs, just for archival purposes?

I took a look at the current configs and they seem ok, but it's possible we're missing something we had in SJC. The web app should be doing the redirect, so it should not be dependent on Apache or php.ini settings as far as I can tell.

Comment 7

7 years ago
I'm reproing this bug again now.

Comment 8

7 years ago
This happened over the weekend randomly and then cleared again. Note, that I've already updated the configs to force https, so that didn't seem to fix it.
(Reporter)

Comment 9

7 years ago
(In reply to comment #2)
> Kohana tries to match a regex to any GET, POST or cookie which causes this
> message:
> 
> http://code.google.com/p/socorro/source/browse/tags/releases/1.7.6_r2892_20110121/webapp-php/system/libraries/Input.php#391
> 
> As far as I know this has only been reproduced on http not https, it is
> possible that this was an existing bug that we have not seen because we were
> consistently redirecting to https before.

Does not seem to have anything to do with HTTP versus HTTPS, and probably not caching related since nagios noticed the problem on each individual web server.

I think the next step should be to add some debug logging to the clean_input_keys() function, and see exactly what it's failing on. 

The nagios check does not use cookies AFAIK and does the same GET over and over, so I can't imagine what is triggering this regex.

Comment 10

7 years ago
FYI: we didn't actually prove that zeus cache isn't the problem. the http checks on the webheads hit the webhead, get redirected to https://crash-stats.mozilla.com/ and therefore hit the zeus, which is why all 5 of them paged at the same time...

Comment 11

7 years ago
I turned off caching in zeus just now.
(In reply to comment #11)
> I turned off caching in zeus just now.

Our Socorro tests have consistently been passing, since: http://qa-selenium.mv.mozilla.com:8080/job/socorro/.  I'll keep an eye on them.
(Assignee)

Updated

6 years ago
Component: Socorro → General
Product: Webtools → Socorro

Updated

6 years ago
Component: General → Webapp

Updated

6 years ago
Flags: in-testsuite?
Fixed by Django
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.