713795 - emails to [e-mail] not getting through (or being very delayed?)

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

I've just talked on phone with legneato, who reports sending a "go to build 10.0b2" email to release-drivers at 18:09, but which RelEng/release-drivers have not yet received at 19:02.

RelEng will start the builds now that we've been verbally told, but this repeat blockage needs to be fixed, so that QA will know when builds are available for testing, and other groups know as part of coordinating the FF10.0b2 release.


If it helps debugging, legneato's email is below:
> From: Christian Legnitto <[e-mail]>
> Date: December 27, 2011 6:09:02 PM PST
> To: release-drivers drivers <[e-mail]>
> Subject: Not Urgent: Please build Firefox 10 (beta 2, build #1)
> content-type: text/plain; charset=us-ascii
> content-transfer-encoding: 7bit
> x-smtp-server: smtp.mozilla.org:[e-mail]
> x-universally-unique-identifier: be7132e4-a3b1-4897-971f-1a9c3bafecf6
> message-id: <[e-mail]>
> mime-version: 1.0 (Apple Message framework v1082)
>
> Please build Firefox 10 (beta 2, build #1) using this changeset:
>
>    https://hg.mozilla.org/releases/mozilla-beta/rev/9588ec86d6e6
>
> Please note this is not an urgent request.
>
> Thanks,
> Christian


Also, a nagios alert about mailman, which might be related.
-------- Original Message --------
Subject: ** PROBLEM alert - dm-mailman01/mailman is CRITICAL **
Date: Tue, 27 Dec 2011 18:09:40 -0800 (PST)
From: [e-mail] (nagios)
To: [e-mail],[e-mail]
***** Nagios  *****
Notification Type: PROBLEM
Service: mailman
Host: dm-mailman01
Address: 10.2.74.23
State: CRITICAL
Date/Time: 12-27-2011 18:09:40
Additional Info:
in has 28238 tasks, oldest 1.4 hrs

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

This is not specific to legneato; an email I sent to release-drivers at 18:48 has not yet been delivered at 19:17.

Comment 2

[Phong Tran [:phong]]

The nagios alert worked just as intended.  We had another AMO outage that caused a bunch of amo-traceback emails to be sent out and backed up the mailman queue.  Someone is looking clearing it out now.

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to Phong Tran [:phong] from comment #2)
> The nagios alert worked just as intended. 
Yep, very cool. First time seeing that since it was setup, so good to know. 

> We had another AMO outage that
> caused a bunch of amo-traceback emails to be sent out and backed up the
> mailman queue.  Someone is looking clearing it out now.

19:38:45 < justdave> you can thank AMO
19:38:56 < justdave> 38K tracebacks mailed to the amo-tracebacks mailing list
19:39:41  * justdave nukes them all
19:40:09  * joduinn sharpens justdave 's axe and gives it back to him
19:40:18 < bkero> justdave: thanks

Comment 4

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

I am now receiving the earlier missing emails to rel-drivers. Also nagios seems happier, so downgrading to normal. 



-------- Original Message --------
Subject: ** RECOVERY alert - dm-mailman01/mailman is OK **
Date: Tue, 27 Dec 2011 19:48:51 -0800 (PST)
From: [e-mail] (nagios)
To: [e-mail],[e-mail]
***** Nagios  *****
Notification Type: RECOVERY
Service: mailman
Host: dm-mailman01
Address: 10.2.74.23
State: OK
Date/Time: 12-27-2011 19:48:51
Additional Info:
all normal

Comment 5

[Dave Miller [:justdave]]

AMO managed to mailbomb the amo-tracebacks mailing list, there were over 38K messages in the queue destined for that list.

Those all got nuked out of the queue, and bug 713800 has been filed to find some other way to deliver those tracebacks.