519579 - Set up buildbot etc on bm-xserve21 after reimage

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

So here's the tricksyness - bm-xserve21 was the second unit test box on the Firefox3.0 tree, and I'm not sure we have great/any docs about how to set up a box to match the other machine (bm-xserve20). Plus, we could use the CPU resources for fx3.5 and later. So here's a couple of options of branch drivers to consider

1) We want to keep two mac unit test boxes on Firefox3.0. In that case we'd take a downtime on bm-xserve20 and clone it across to bm-xserve21. I think we should only do this if there is significant flakiness in mac unittests, and we therefore need a second box to help decide if a failure is intermittent or not. Recent history is perma-green.

2) We move bm-xserve21 into the slave pool for Fx3.5 and later. It's already imaged appropriately, and just need to set up puppet and buildbot. This would help with wait times, particularly on m-c and m-1.9.2, which I would say is a higher priority right now.

What do you say Sam ?

Comment 2

[Samuel Sidler (old account; do not CC)]

I'm fine with moving bm-xserve21 to the slave pool for Fx3.5 and later. Note that I think we should take an image of bm-xserve20 anyway so we have a backup "just in case". Hopefully we won't need it in the next few months, but better safe than sorry, right?

Sorry for the delay in responding. I had the tab open, but it got lost.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Thanks Sam. Having a backup is a fine idea, filed bug 522619.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Fixing up nagios is bug 522627.

Comment 5

[Lukas Blakk [:lsblakk] use ?needinfo]

bm-xserve21 is now attached to production-master02

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

bug 522627 added the checks on the nagios server, but some most of them were failing - average_load, buildbot, and root partition. The ping check was working but that doesn't rely on nagios functioning on bm-xserve21.

Borrowing 
  sudo /usr/local/nagios/sbin/enablenrpe
from the 10.5 ref doc fixed it up, but the original problem may indicate something is busted in puppet.

Comment 7

[[e-mail] (:bhearsum)]

(In reply to comment #6)
> bug 522627 added the checks on the nagios server, but some most of them were
> failing - average_load, buildbot, and root partition. The ping check was
> working but that doesn't rely on nagios functioning on bm-xserve21.
> 
> Borrowing 
>   sudo /usr/local/nagios/sbin/enablenrpe
> from the 10.5 ref doc fixed it up, but the original problem may indicate
> something is busted in puppet.

Weird, this command gets run through Puppet, http://hg.mozilla.org/build/puppet-manifests/file/871a77bc10e8/os/osx.pp#l155