836729 - ISE 500 errors from ftp cluster yesterday

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[[e-mail] (:bhearsum)]

We had a bunch of ISE 500 errors from the FTP cluster yesterday around 18:55 PT. I _think_ the errors may have actually come at the Zeus level, because we got output like this:
Executing: ['wget', '--progress=dot:mega', '--server-response', '--no-check-certificate', '-O', 'update/complete.mar', 'http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/19.0b4-candidates/build1/update/mac/ach/firefox-19.0b4.complete.mar']
--18:56:33--  http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/19.0b4-candidates/build1/update/mac/ach/firefox-19.0b4.complete.mar
           => `update/complete.mar'
Resolving stage.mozilla.org... 63.245.215.47
Connecting to stage.mozilla.org|63.245.215.47|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 500 Internal Server Error
  Date: Thu, 31 Jan 2013 02:56:33 GMT
  Connection: close
  Content-Type: text/html
18:56:33 ERROR 500: Internal Server Error.

Where normally we would have more headers, including X-Backend-Server.

Could this be related to turning ftp5 back on in bug 792848?

Comment 1

[Dumitru Gherman [:dumitru]]

Yeah, see bug 826495 for reasons why this has happened.
I was making some changes yesterday around that time to an Apache config file on that cluster, and Apache didn't restart gracefully.
:jakem did some awesome troubleshooting last night, and we'll have some fixes soon.
Closing this now.