888785 - AWS masters aren't deleting old builds, need to install procmail package

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

The cron tries to make a lock file using ... wait for it ... lockfile. That's provided by the procmail package, which isn't installed on AWS masters. So we end up getting Nagios alerts like
 buildbot-master73.srv.releng.usw2.mozilla.com:disk - / is WARNING: DISK WARNING - free space: / 10464 MB

Do we just need to add 
       include packages::mozilla::procmail
to puppet/modules/buildmaster/manifests/buildbot_master/{simple,mozilla}.pp ?

We have data going back to April 11 on bm73. As it's only just filling the 100G disk we could consider increasing the '-t4' argument for master_cleanup.py, see http://hg.mozilla.org/build/puppet/file/d53bbd8d4736/modules/buildmaster/templates/buildmaster-cron.erb
Might need to parametrize that though, because the panda masters still in scl1 have much smaller disks. Also much less traffic, so maybe it works out OK.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Cron may also be failing to send mail. I can't find anything complaining of lockfile not being found.

Comment 2

[Justin Wood (:Callek)]

(In reply to Nick Thomas [:nthomas] from comment #1)
> Cron may also be failing to send mail. I can't find anything complaining of
> lockfile not being found.

So it still sends mail, afaict (did a bit of quick testing)

It seems to not send the mail about lockfile because we pipe its stderr to /dev/null and cron won't send mail if there is no output.

Comment 3

[Justin Wood (:Callek)]

Attachment: [puppet] add procmail

This adds procmail to both mozilla masters and servo master.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 769571 [details] [diff] [review]
[puppet] add procmail

Wasn't in a position to write a patch, so certainly not going to review.

Comment 5

[Justin Wood (:Callek)]

Comment on attachment 769571 [details] [diff] [review]
[puppet] add procmail

https://hg.mozilla.org/build/puppet/rev/d2417bf55b99