jst reported a problem in #socorro around 9:03 this morning: 17:03 < jst> so I was trying to load http://crash-stats.mozilla.com/products/Firefox 17:04 < jst> and I get a page that says "Disallowed key characters in global data." lars suggested clearing cookies, which seems to have fixed for jst. lars could also reproduce this (http only, not with https) and clearing cookies seems to have helped. This is not an expected error (we only saw it in SJC when it was in a really bad state, postgres was down. There is an "orange screen" kohana error related to cookies when going from 1.7.6 to an earlier version, but we don't think this is it). stephend reproduced this at 9:07, with user agent 3.6.13 on NT 6.1. I haven't seen an error around this time, going to try to track it down in the access logs. I haven't found anything untoward in the logs. We want to: 1) make sure this is not a result of traffic going to SJC (it should not be at this point, the old loadbalancer is sending all traffic to PHX) 2) understand why the error is happening in PHX, if that's the case One thing ryansnyder noticed is that the error happened on a http URL, but we should be redirecting those consistently to https. May or may not be related, there will be a separate bug for that.
(In reply to comment #0) > lars suggested clearing cookies, which seems to have fixed for jst. lars could > also reproduce this (http only, not with https) and clearing cookies seems to > have helped. I saw this last night, too (I forget at what time, and I forgot to look for debug output on my end) trying to load http://crash-stats.mozilla.com/ in a completely fresh profile (so no cookies at all) when I was testing a patch. Switching to https loaded correctly.
Kohana tries to match a regex to any GET, POST or cookie which causes this message: http://code.google.com/p/socorro/source/browse/tags/releases/1.7.6_r2892_20110121/webapp-php/system/libraries/Input.php#391 As far as I know this has only been reproduced on http not https, it is possible that this was an existing bug that we have not seen because we were consistently redirecting to https before.
I have a hunch that switching to https will fix the issue. We have a few cookies that are https only, which made me wonder if that was a contributing or root cause of the Disallowed keys error.
If we were consistently redirecting to https before, let's do that here as well. Let's do it anyway, I'm pretty sure the infra can cope with the extra load.
David, we need to change baseurl to point to https:// (http://viewvc.svn.mozilla.org/vc/projects/socorro_qa/vars.py?revision=81228&view=markup). rhelmer also suggests we "put a check in there to make sure http:// gets redirected to https://" after the initial GET request.
OS: Linux → All
Hardware: x86_64 → All
Jabba, can we compare PHX web configs against the SJC configs? An easy way to do this is to drop the SJC ones into prod/etc-socorro/web/ and run "svn diff". Maybe we should find a place in puppet's SVN to check in the old configs, just for archival purposes? I took a look at the current configs and they seem ok, but it's possible we're missing something we had in SJC. The web app should be doing the redirect, so it should not be dependent on Apache or php.ini settings as far as I can tell.
I'm reproing this bug again now.
This happened over the weekend randomly and then cleared again. Note, that I've already updated the configs to force https, so that didn't seem to fix it.
FYI: we didn't actually prove that zeus cache isn't the problem. the http checks on the webheads hit the webhead, get redirected to https://crash-stats.mozilla.com/ and therefore hit the zeus, which is why all 5 of them paged at the same time...
I turned off caching in zeus just now.
(In reply to comment #11) > I turned off caching in zeus just now. Our Socorro tests have consistently been passing, since: http://qa-selenium.mv.mozilla.com:8080/job/socorro/. I'll keep an eye on them.
Component: Socorro → General
Product: Webtools → Socorro
Fixed by Django
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.