662423 - mozilla.com stage as well as other sites on VMs are down

Status: RESOLVED DUPLICATE of bug 662194

(mozilla.org Graveyard :: Server Operations, task)

Description

[Pascal Chevrel (relman team) -> :pascalc]

http://www.stage.mozilla.com/ gives me a "service unavailable" message, (trunk and production are fine)

Comment 1

[Pascal Chevrel (relman team) -> :pascalc]

http://granary.stage.mozilla.com/ , http://l10n-misc.stage.mozilla.com/ , http://google_snippets.stage.mozilla.com/ are also down, raising severity as this is blocking work for myself and localizers.

It looks like all VMs starting as *.stage.mozilla.com are down

Comment 2

[Jake Maul [:jakem]]

We are aware of this issue. It appears to be caused by a problem with input.stage.mozilla.com, which is overloading the server that www.stage.mozilla.com is using. Once that issue is resolved, this one will be as well.

Comment 3

[Pascal Chevrel (relman team) -> :pascalc]

thanks for the quick answer Jake

Comment 4

[Ashish Vijayaram [:ashish]]

This is stable now (details in bug:662194 and bug:662412). Dropping severity (bump up if required) but shall leave this open for tracking.

Comment 5

[Jake Maul [:jakem]]

We have a game plan of stopping the input.stage site overnight if the problem is not solved by then. During the day, I'll be watching the load so I can head the problem off.

In the meantime, devs are working on a fix. Check out the bugs this one depends on.

I'm going to close this bug out as a dupe of 662194. If anyone feels this is in error, please let me know and/or reopen it. Thanks!