654279 - adjust destination monitoring host in runslave.py's buildbot-start reporting

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

The passive buildbot-start check is working *great* for hosts in mtv1 and sjc1, since it's reporting to the correct nagios host (bm-admin01.mozilla.org).

It isn't working so well for the hosts in scl1, since their checks are not handled on that host.

There are two things to do to fix this:

1. Reconfigure nsca on dm-nagios01 to use XOR 'encryption', since that's all runslave.py supports.

2. Add code to runslave to determine the current datacenter (via reverse DNS) and select a monitoring host appropriately.  A better long-term solution is to use DNS scopes to resolve an unqualified hostname based on the DNS search path from DHCP -- but those search paths are not currently set up to support such a solution.  Amy's filing a bug to set that up.

Assigning initially to Amy in hopes that (1) is nice and easy - bump it over to me after that.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

My bad - the scl1 hosts should be pointed to admin1.scl1.infra.mozilla.com, which is already accepting XOR.  Let it not be said releng is the only group with confusing hostnames :)

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: m654279-puppet-manifests-r1.patch

I'll open a new bug for the longer-term solution mentioned in (2)

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

changeset:   322:ffb5425b8ffd
tag:         tip
user:        Dustin J. Mitchell <[e-mail]>
date:        Thu May 05 15:36:57 2011 -0500
summary:     Bug 654279: special-case runslave.py for dc-specific nagios servers; r=catlee

New bug is bug 655136.