762227 - Mail is piling up on cruncher

Status: RESOLVED INVALID

(Infrastructure & Operations :: RelOps: General, task)

Description

[Chris Cooper [:coop] (he/him)]

The briar-patch emails from cruncher stopped being delivered on May 29. I just looked at the mailq on cruncher and it's reporting 843 requests pending, of which most look like this:

(delivery temporarily suspended: connect to cruncher.build.mozilla.org[10.26.48.43]:25: Connection refused)

These briar-patch emails are a big crutch for buildduty. Can we please have someone look at the mail config on cruncher?

Comment 1

[Amy Rich [:arr] [:arich]]

The problem appears to be that the host (running postfix) is trying to send mail to cruncher.build.mozilla.org, which is not the machine's local hostname.  So it does a DNS lookup and finds the IP, tries to connect to it and can't (because postfix only runs on 127.0.0.1).

I would think that, ideally, we'd be using a smarthost, but smtp.mozilla.com doesn't accept connections, either (I briefly tried setting relayhost and flushing the queue).

I also see that mail is being sent to addresses outside of mozilla, and this definitely will not function because cruncher.build.mozilla.org is not a valid hostname outside of mozilla.  If we're sending mail to people outside of mozilla, we should probably be masquerading.

I'm handing this off to justdave, since I don't know what changed on May 29th, my experience is more with sendmail than postfix, and I'm not sure what the best recommended fix for this might be given the above issues.

Comment 2

[Dave Miller [:justdave]]

You *should* be using smtp.mozilla.org as a relay host/smarthost.  If that's not working something is broken.  If it's getting timeouts then it's probably a netflow issue.

Comment 3

[Dave Miller [:justdave]]

If it's trying to deliver mail *to* cruncher, you probably need to tell postfix that it owns that domain name so it'll just process it instead of trying to send it out.

Comment 4

[Amy Rich [:arr] [:arich]]

It looks like most of those were bounce messages (I'm guessing) trying to be delivered to [e-mail] (which is not a valid mail host).  

I have added smtp.mozilla.org and flushed the queue at this point and there's nothing left.

cruncher.build.mozilla.org is still an invalid from address if that's what mail is being sent out as, so all hosts external to mozilla will reject the mail.

How can I test if this is working for the intended purpose, coop?

Comment 5

[Amy Rich [:arr] [:arich]]

Er, sorry, that should have been [e-mail].

Comment 6

[Mike Taylor [:bear]]

I manually triggered a job that generates emails - will post the results when it's done

Comment 7

[Chris Cooper [:coop] (he/him)]

(In reply to Amy Rich [:arich] [:arr] from comment #4)
> It looks like most of those were bounce messages (I'm guessing) trying to be
> delivered to [e-mail] (which is not a valid mail host).  

That looks like something broken on our end. Maybe a crontab setting MAILTO poorly. I'll investigate.
 
> cruncher.build.mozilla.org is still an invalid from address if that's what
> mail is being sent out as, so all hosts external to mozilla will reject the
> mail.
> 
> How can I test if this is working for the intended purpose, coop?

As long as I can send mail to @mozilla.com address and aliases, that should be fine. I tested and I can send mail to [e-mail], so this is fixed enough for me.

Comment 8

[Amy Rich [:arr] [:arich]]

So it turns out that the reason that briar-patch  was not sending out mail was... that the code to send out mail had been commented out.