432219 - Email notification for mobile build failures

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Christian Sejersen]

Comment 1

[Christian Sejersen]

I would like to get an email notification when any of the mobile builds fail. 

Comment 2

[Benjamin Smedberg]

It would be a lot less work if you could just use the quickparse data from tinderbox to determine this and take action (like firebot does in IRC), rather than setting up another release-team task.

Comment 3

[Benjamin Smedberg]

And this definitely isn't build-config ;-)

Comment 4

[[e-mail] (:bhearsum)]

No worries, I asked him to file it.

Comment 5

[Christian Sejersen]

Instead of emailing me, can you use this email address please: [e-mail]

Comment 6

[Benjamin Smedberg]

Are you sure you want to email the newsgroup every time the build goes red? Builds going red (not just mobile) mhappens frequently enough that it sounds like it would be unwelcome spam on the list.

Comment 7

[Christian Sejersen]

I am aware that there is a spam risk, but yes, I would like to try it out. 

Comment 8

[[e-mail] (:bhearsum)]

Attachment: send e-mail to [e-mail] on failure

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 320547 [details] [diff] [review]
send e-mail to [e-mail] on failure

Looks fine to me, except that the modes are defined
 *  'failing': only send mail about builds which fail
 *  'problem': only send mail about a build which failed when the previous build passed
So 'failing' will ensure lots of messages to the mailing list if mobile is broken for a spell, while 'problem' gives you just the one. Your call Christian, go ahead check it in either way.

Comment 10

[Christian Sejersen]

I would prefer to get email in the 'problem' case only. 

Comment 11

[[e-mail] (:bhearsum)]

Attachment: [checked in] patch as landed

Checked in as changeset:   55:0defef3c8d9e.

You should receive e-mails for "problems" from now on.

Comment 12

[Christian Sejersen]

I haven't received any emails yet even thought the build has has "problems" a few times.

Comment 13

[[e-mail] (:bhearsum)]

Hm, that's odd. Let me check the logs.

Comment 14

[Christian Sejersen]

Did you find anything Ben?

Comment 15

[[e-mail] (:bhearsum)]

Before I pass this off to IT can someone on the mailing list look for e-mail from ([e-mail] [w/ the typo]) on May 16th at 17:18 PDT. Please check your spam bin too.

I think it's getting spammed either because of the content, or the typo in the fromaddr (which I just fixed).

Comment 16

[Christian Sejersen]

I asked everybody at the mobile meeting yesterday and nobody has seen any build mails. 

I have checked my junk email as well and no sign of it. Is it possible the mailing list marks it as spam and doing so doesn't forward it?

Comment 17

[[e-mail] (:bhearsum)]

That's entirely possible, yeah. I'm going to pass this to IT to see if they can check mail server logs.

IT:
Can you look at the mail server logs for e-mail detailed in comment#15? It was sent through mail.build.mozilla.org.

Comment 18

[matthew zeier [:mrz]]

Looks like it passed through:

May 16 17:18:01 dm-mail02 amavis[31140]: (31140) AM.PDP  /var/amavis/afXXXXZYtIn4: <[e-mail]> -> <[e-mail]>

May 16 17:18:01 dm-mail02 amavis[31140]: (31140) Checking: JItPr-V0fLqB AM.PDP-SOCK [10.2.71.90] <[e-mail]> -> <[e-mail]>

May 16 17:18:01 dm-mail02 amavis[31140]: (31140) SPAM-TAG, <[e-mail]> -> <[e-mail]>, No, score=-4.398 required=6.31 tests=[ALL_TRUSTED=-1.8, BAYES_00=-2.599, WEIRD_PORT=0.001]

May 16 17:18:01 dm-mail02 amavis[31140]: (31140) Passed CLEAN, AM.PDP-SOCK LOCAL [10.2.71.90] [10.2.71.90] <[e-mail]> -> <[e-mail]>, Queue-ID: 40CC78240D0, Message-ID: <[e-mail]>, mail_id: JItPr-V0fLqB, Hits: -4.398, size: 965, 392 ms

May 16 17:18:01 dm-mail02 postfix/qmgr[2980]: 40CC78240D0: from=<[e-mail]>, size=1019, nrcpt=1 (queue active)


and looks like it was forwarded off:

May 16 17:18:22 dm-mail02 postfix/smtp[31283]: 40CC78240D0: to=<[e-mail]>, relay=lists.mozilla.org[10.2.74.113]:25, delay=21, delays=0.49/0.01/0/21, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 0169D11F5C)
May 16 17:18:22 dm-mail02 postfix/qmgr[2980]: 40CC78240D0: removed

Comment 19

[matthew zeier [:mrz]]

Honestly have no idea how to debug mailman -> usenet, pushing to Dave.

Comment 20

[Dave Miller [:justdave]]

"Post by non-subscriber to a members-only list".

It was stuck in the moderator queue.  I cleared all of the messages in the queue and added that address to the "automatically accept mail from this address" list.

Being that this bug looks like it's for a larger project, sending back to the original assignee instead of resolving...

Comment 21

[Dave Miller [:justdave]]

and FYI, the bounce message saying it had been held for the moderator bounced...

May 16 17:18:25 notorious postfix/smtp[23577]: B3EA811F5C: to=<[e-mail]>, relay=build.mozila.org[69.50.169.251]:25, delay=0.9, delays=0.65/0/0.18/0.07, dsn=5.7.1, status=bounced (host build.mozila.org[69.50.169.251] said: 550 5.7.1 <[e-mail]>... Relaying denied (in reply to RCPT TO command))

Comment 22

[[e-mail] (:bhearsum)]

Yeah, I typoed the from address, it's fixed now. Can you make sure '[e-mail]' is on the list, too?

Also, Christian: Do you really want Buildbot failures to go out to the newsgroups?

Comment 23

[Christian Sejersen]

We are getting the failure notifications now but in the email this link (http://staging-master.build.mozilla.org:8020/builders/mobile-linux-arm-dep/builds/632) gives a 404.

Comment 24

[[e-mail] (:bhearsum)]

Those links only work if you have mpt-vpn access. Really, you should be checking the Tinderbox page when you see a failure. (If you _do_ have mpt-vpn access that'll work generally, but it's probably down right now.)

Comment 25

[Christian Sejersen]

So we don't have any notifications that will link to the tinderbox page? I am just asking since we have several non-MoCo people who will be notified.

Comment 26

[[e-mail] (:bhearsum)]

Unfortunately there's no way to customize the messages yet :(.