659523 - crash-stats is returning error page

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Daniel Veditz [:dveditz]]

http://crash-stats.mozilla.com/ is returning an error page saying it's the site, not me -- but I'm surprised not to see a bug report on it.

Comment 1

[Daniel Veditz [:dveditz]]

It was working OK for a while: couldn't find a particular signature in the default 5.0 topcrash report (from the dropdown) so I searched in Firefox 4.0.1 and got an error trying to load the topcrash report. Now I get an error page for the main page.

Comment 2

[Robert Helmer [:rhelmer]]

This is due to http://blog.mozilla.com/socorro/2011/05/24/configuration-changes-may-24/ and should be resolved shortly.

Comment 3

[Daniel Veditz [:dveditz]]

apparently missed the link to that on dev-planning, thanks.

ObNit: that post says we might get 404s, which is not what you actually see.

Comment 4

[Robert Helmer [:rhelmer]]

(In reply to comment #3)
> apparently missed the link to that on dev-planning, thanks.
> 
> ObNit: that post says we might get 404s, which is not what you actually see.

Yes we actually hit an unexpected problem; this was a routine DB failover test, and the web heads can't reconnect although everything else can. Working on bringing it back up, will add more info asap :)

Comment 5

[Robert Helmer [:rhelmer]]

Posted a comment to the blog with a little more info.

Comment 6

[Robert Helmer [:rhelmer]]

(In reply to comment #3)
> apparently missed the link to that on dev-planning, thanks.

We've been updating the blog consistently, but we could certainly post to dev-planning as well. 

This was supposed to be a routine database failover test, good thing that we found this but still, always chance of downtime.

We have been drawing a distinction between the more critical collection, processing functions and the frontend functions in terms of downtime notice, etc.

Comment 8

[Robert Helmer [:rhelmer]]

We're back up, and the problem (Zeus loadbalancer configuration) has been corrected, so we're in a better place in case of an unplanned failover. However we will need to fail back to the primary database sometime in the near future, we expect that to go without issue now.