Closed Bug 632285 Opened 13 years ago Closed 13 years ago

SUMO returning 500 and empty page over HTTPS

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Platform:
All
Other
Type:
task
Priority:
Not set
Severity:
blocker

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jsocol, Assigned: justdave)

References

Details

Was just reported in #sumo and I can confirm that https://support.mozilla.com isn't returning any content and is sending a 500 internal server error.

Not reproducible over HTTP, and no Zeus-style error message, either. Not sure why Nagios hasn't paged yet.
Can't find anything interesting in the error logs at all.  Zeus has all three nodes failed out and says they've been failed for 3 hours.  Where's our monitoring?
I restarted apache on all three nodes, then disabled and re-enabled one of them in Zeus to force-clear it, and it immediately failed out again.
Assignee: server-ops → justdave
The root cause is a changed httpd.conf that was checked in on 1/10 but not pushed out until today.  We are still working on the root cause details, but the changed config file has been reverted and pushed out to the webheads.  SUMO is back up.

More embarrassingly, our monitoring failed to catch this.  We are fixing this immediately.
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Depends on: 632298
Depends on: 632299
Depends on: 632300
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.