755447 - mail.mozilla.org is down.

Status: RESOLVED FIXED

(Infrastructure & Operations :: Infrastructure: Other, task)

Description

[Maire Reavy [:mreavy]]

I'm the administrator for the media-private mailing list, and I can't access the mailing list to approve posts, etc.

I was reviewing the admin pages of media-private and then after clicking on a link I got the following error message:

"Bug in Mailman version 2.1.14
We're sorry, we hit a bug!
Please inform the webmaster for this site of this problem. Printing of traceback and other system information has been explicitly inhibited, but the webmaster can find this information in the Mailman error logs."

Comment 1

[Dave Miller [:justdave]]

The network storage shelf storing the backend filesystem for the mailman server got taken offline for the datacenter move.  It's going to be down until further notice.  We're working on figuring out what to do about it.

Comment 2

[Maire Reavy [:mreavy]]

(In reply to Dave Miller [:justdave] from comment #1)
> The network storage shelf storing the backend filesystem for the mailman
> server got taken offline for the datacenter move.  It's going to be down
> until further notice.  We're working on figuring out what to do about it.

What does this mean for my list?  Can members still send email to media-private (and will the emails get archived), or does this just affect the administration of the list?  Thanks.

Comment 3

[Dave Miller [:justdave]]

The mail server is effectively dead for the time being.  Inbound mail will be queued, and will get delivered after we get the server back online.  It's looking like we might have it back online within the next hour or so.

Comment 4

[Dave Miller [:justdave]]

(In reply to Dave Miller [:justdave] from comment #3)
> The mail server is effectively dead for the time being.  Inbound mail will
> be queued, and will get delivered after we get the server back online.  It's
> looking like we might have it back online within the next hour or so.

That turned out to be a far-fetched idea at the time because too many things broke at once :|

*now* I expect it within an hour.

Comment 5

[Dave Miller [:justdave]]

And this is now a blocker because it's affecting mozilla.org email.

Comment 6

[Dave Miller [:justdave]]

Been ramping up inbound mail to it via relays slowly just to make sure everything was working before unleashing the floodgates.  The queues have all been flushed and caught up now, and it looks like everything's working.  I just pushed the last DNS change to get the web interface to mailman working again, should propagate in 10 or 15 minutes.

Any additional issues that seem related either reopen or flag me down on IRC.  If something seems outright broken, reset assignee to default so it'll page someone.