662194 - [stage] Staging env returns a 500 error

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Matt Brandt [:mbrandt]]

Attachment: Fail message

The staging env appears to be down.

Steps to reproduce:
1. goto http://input.stage.mozilla.com/

Actual:
"Service Unavailable" message is displayed by the site.

Comment 1

[Corey Shields [:cshields]]

mrapp-stage02 is not responding (nor is its out of band).  Phong is en route to check on it.

Comment 2

[Jake Maul [:jakem]]

This server is responding again, and we're looking into what happened.

One thing I can tell you, however, is that there appears to be a problem with http://input.stage.mozilla.com/en-US/. When I attempt to visit that page, the Apache process handling my request shoots up to 100% CPU usage and hangs there.

Comment 3

[Jake Maul [:jakem]]

Dropping prio since the server is working again and this is now "just" a problem with input.stage, plus some investigative work.

Comment 4

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

Let me know if it's something strange on our end.

Comment 5

[Jake Maul [:jakem]]

I do believe it is some type of coding issue, but I can't pinpoint it more specifically that just the URL:

http://input.stage.mozilla.com/en-US/

This feels like it's stuck the server in some type of infinite loop. By that I mean, when someone hits that page, I can watch an Apache worker suddenly jump up to 100% CPU usage and stay there forever, until someone kills it. In a browser, you'll get a Zeus 500 ISE error after 30 seconds or so, but the Apache worker on the server keeps going for as long as 5 minutes (longest I've seen one before it was killed manually).

The main Apache error_log has nothing, and the same goes for input.stage.mozilla.com's error_log. I doubt it's emailing you stack traces or anything, either.


I believe what happened is enough of these got loaded up at once that the server simply became unresponsive entirely, until the kernel "Out of Memory Killer" killed off one of offending httpd processes. This has happened twice yesterday morning, twice this morning, and once this afternoon (1:59:05pm MV time).


Note that other languages behave the same way (http://input.stage.mozilla.com/es/), but that other pages do work (http://input.stage.mozilla.com/en-US/feedback).

Comment 6

[Ashish Vijayaram [:ashish]]

This happened again at ~0815 PDT, host ran out of RAM and swap and had to reboot to get the host back up. It was extremely busy oom-killing processes and took about 20-30 mins to reboot.

Comment 8

[Jake Maul [:jakem]]

Considering this a solved code issue- the features on input.stage.mozilla.com suspected of causing this server-wide issue have been removed, and will not be re-enabled unless reimplemented some other way. Thanks all!

Comment 9

[Matt Brandt [:mbrandt]]

thx for the attention... stage seems to be fantastic again. No longer going down with ISEs