SUMO returning 500 and empty page over HTTPS

RESOLVED FIXED

Status

--
blocker
RESOLVED FIXED
8 years ago
4 years ago

People

(Reporter: jsocol, Assigned: justdave)

Tracking

Details

(Reporter)

Description

8 years ago
Was just reported in #sumo and I can confirm that https://support.mozilla.com isn't returning any content and is sending a 500 internal server error.

Not reproducible over HTTP, and no Zeus-style error message, either. Not sure why Nagios hasn't paged yet.
Can't find anything interesting in the error logs at all.  Zeus has all three nodes failed out and says they've been failed for 3 hours.  Where's our monitoring?
I restarted apache on all three nodes, then disabled and re-enabled one of them in Zeus to force-clear it, and it immediately failed out again.

Updated

8 years ago
Assignee: server-ops → justdave
The root cause is a changed httpd.conf that was checked in on 1/10 but not pushed out until today.  We are still working on the root cause details, but the changed config file has been reverted and pushed out to the webheads.  SUMO is back up.

More embarrassingly, our monitoring failed to catch this.  We are fixing this immediately.
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
Depends on: 632298
Depends on: 632299
Depends on: 632300
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.