656415 - hosts in DNS that we don't monitor

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Amy Rich [:arr] [:arich]]

After weeding out DNS/DHCP, I've found the following hosts in the build domain in mtv1 and sjc1 that do not appear to be monitored.  We should probably do so for most of them, at least, if only to tell when the hardware fails.

I suspect that some of these may be monitored in some other way in a name I don't know about (like the hosts named pdu*), and we shouldn't also monitor them with bm-admin01.

Which hosts should we monitor, and what checks should we use?  Should they be in any hostgroups?

admin.build.sjc1.mozilla.com.
bm-foopy.build.mtv1.mozilla.com.
bm-mini-build-ref.build.sjc1.mozilla.com.
bm-parallels01.build.sjc1.mozilla.com.
bm-remote.build.mtv1.mozilla.com.
bm-symbolfetch01.build.sjc1.mozilla.com.
bm-win2k3-test.build.sjc1.mozilla.com.
boris.build.sjc1.mozilla.com.
bv-vmware1.build.mtv1.mozilla.com.
byob-repack01.build.mtv1.mozilla.com.
byob-repack02.build.mtv1.mozilla.com.
byob-repack03.build.mtv1.mozilla.com.
byob-repack04.build.mtv1.mozilla.com.
byob-repack05.build.mtv1.mozilla.com.
cb-vmware1.build.sjc1.mozilla.com.
cm-bbot-leopard-001.build.sjc1.mozilla.com.
cm-bbot-leopard-002.build.sjc1.mozilla.com.
cm-bbot-leopard-003.build.sjc1.mozilla.com.
deploystudio.build.sjc1.mozilla.com.
deploystudiopcserver.build.mtv1.mozilla.com.
foopy01.build.mtv1.mozilla.com.
foopy02.build.mtv1.mozilla.com.
foopy03.build.mtv1.mozilla.com.
foopy04.build.mtv1.mozilla.com.
fx-win32-real-hw.build.sjc1.mozilla.com.
linux-ix-ref.build.mtv1.mozilla.com.
linux-ref-platform.build.sjc1.mozilla.com.
linux64-ref-platform.build.sjc1.mozilla.com.
mac-keymaster.build.sjc1.mozilla.com.
mobile-image03.build.mtv1.mozilla.com.
moz2-darwin10-ref.build.mtv1.mozilla.com.
moz2-linux-workstation.build.sjc1.mozilla.com.
mw64-ix-slave01.build.mtv1.mozilla.com.
nm-ops02.build.sjc1.mozilla.com.
nm-ops03.build.sjc1.mozilla.com.
pdu1.df201-4.build.mtv1.mozilla.com.
pdu2.df201-4.build.mtv1.mozilla.com.
preed-win32-test.build.sjc1.mozilla.com.
t-r3-w764-ref.build.mtv1.mozilla.com.
talos-r3-fed-ref.build.mtv1.mozilla.com.
talos-r3-fed64-ref.build.mtv1.mozilla.com.
talos-r3-leopard-ref.build.mtv1.mozilla.com.
talos-r3-snow-ref.build.mtv1.mozilla.com.
talos-r3-w7-ref.build.mtv1.mozilla.com.
talos-r3-xp-ref.build.mtv1.mozilla.com.
tintin.build.mtv1.mozilla.com.
v90-boris.build.sjc1.mozilla.com.
v90-nm-dhcp1.build.sjc1.mozilla.com.
win2k3-ref-img.build.sjc1.mozilla.com.
win32-ix-ref.build.mtv1.mozilla.com.
win64-ix-ref.build.mtv1.mozilla.com.

Comment 1

[Mike Taylor [:bear]]

(In reply to comment #0)

> foopy01.build.mtv1.mozilla.com.
> foopy02.build.mtv1.mozilla.com.
> foopy03.build.mtv1.mozilla.com.
> foopy04.build.mtv1.mozilla.com.

android tegra test servers

> bm-foopy.build.mtv1.mozilla.com.

tegra staging server

Comment 2

[Aki Sasaki (not active)]

(In reply to comment #1)
> android tegra test servers

We should monitor the foopy's for ping/disk/cpu plus some processes, but depending on the amount of work and how fast we go to new foopy's it may or may not be worth the work to do that now.

pdu1 and 2 are probably powering the tegras. Not sure how to monitor these, but it would be good.  Ping checks? web checks?

bm-remote.build.mtv1.mozilla.com is the load-balancer ip for bm-remote-talos-webhost-0{1,2,3}.  Since those aren't on the list, I guess those are being monitored?  If so, just knowing that bm-remote is pingable and serving port 80 is probably fine.

cm-bbot* are Dustin's buildbot community boxes. I'll let him answer there.

The *ref* boxes probably don't need any monitoring.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

In general, I think that everything we expect to be up should at least be ping'd.  Aside from what's been mentioned above:

(In reply to comment #0)
> admin.build.sjc1.mozilla.com.
  PING

> bm-mini-build-ref.build.sjc1.mozilla.com.
> t-r3-w764-ref.build.mtv1.mozilla.com.
> talos-r3-fed-ref.build.mtv1.mozilla.com.
> talos-r3-fed64-ref.build.mtv1.mozilla.com.
> talos-r3-leopard-ref.build.mtv1.mozilla.com.
> talos-r3-snow-ref.build.mtv1.mozilla.com.
> talos-r3-w7-ref.build.mtv1.mozilla.com.
> talos-r3-xp-ref.build.mtv1.mozilla.com.
> linux-ix-ref.build.mtv1.mozilla.com.
> win2k3-ref-img.build.sjc1.mozilla.com.
> win32-ix-ref.build.mtv1.mozilla.com.
> win64-ix-ref.build.mtv1.mozilla.com.
> moz2-darwin10-ref.build.mtv1.mozilla.com.
  PING (then we'd have known earlier that t-r3-w764-ref is dead)

> byob-repack01.build.mtv1.mozilla.com.
> byob-repack02.build.mtv1.mozilla.com.
> byob-repack03.build.mtv1.mozilla.com.
> byob-repack04.build.mtv1.mozilla.com.
> byob-repack05.build.mtv1.mozilla.com.
> mac-keymaster.build.sjc1.mozilla.com.
  coop?

> bm-parallels01.build.sjc1.mozilla.com.
> bm-symbolfetch01.build.sjc1.mozilla.com.
> bm-win2k3-test.build.sjc1.mozilla.com.
> boris.build.sjc1.mozilla.com.
> bv-vmware1.build.mtv1.mozilla.com.

> cb-vmware1.build.sjc1.mozilla.com.
> fx-win32-real-hw.build.sjc1.mozilla.com.
> linux64-ref-platform.build.sjc1.mozilla.com.
> moz2-linux-workstation.build.sjc1.mozilla.com.
> mw64-ix-slave01.build.mtv1.mozilla.com.
  I personally no idea what these are.  Rather than figure it out, let's not monitor them :)

> mobile-image03.build.mtv1.mozilla.com.
  laptop - no monitoring required

> cm-bbot-leopard-001.build.sjc1.mozilla.com.
> cm-bbot-leopard-002.build.sjc1.mozilla.com.
> cm-bbot-leopard-003.build.sjc1.mozilla.com.
> preed-win32-test.build.sjc1.mozilla.com.
> tintin.build.mtv1.mozilla.com.
  remove (as discussed)

> v90-boris.build.sjc1.mozilla.com.
> v90-nm-dhcp1.build.sjc1.mozilla.com.
> nm-ops02.build.sjc1.mozilla.com.
> nm-ops03.build.sjc1.mozilla.com.
> deploystudio.build.sjc1.mozilla.com.
> deploystudiopcserver.build.mtv1.mozilla.com.
  infra - your call

Comment 4

[Chris Cooper [:coop] (he/him)]

(In reply to comment #3)
> > byob-repack01.build.mtv1.mozilla.com.
> > byob-repack02.build.mtv1.mozilla.com.
> > byob-repack03.build.mtv1.mozilla.com.
> > byob-repack04.build.mtv1.mozilla.com.
> > byob-repack05.build.mtv1.mozilla.com.

PING check for now. buildbot check eventually.

> > mac-keymaster.build.sjc1.mozilla.com.

No one has picked up Syed's signing-on-mac work. This machine is not currently important, but will be at some point, in theory. Ping monitoring for now.

> > cb-vmware1.build.sjc1.mozilla.com.

Does this machine host the VMs we host for community projects like SeaMonkey. Probably needs ping coverage, but maybe monitored elsewhere already?

> > fx-win32-real-hw.build.sjc1.mozilla.com.

Was used for VM vs. hardware build time testing. Can be recycled IMO if it still exists and isn't just an alias for another machine.

> > linux64-ref-platform.build.sjc1.mozilla.com.

Is this our ref image for Linux64, or just a poorly-named facsimile thereof meant to confuse us?

> > mw64-ix-slave01.build.mtv1.mozilla.com.

This is our lone win64 builder AIUI. Should have ping and buildbot monitoring AFAICT. Armen?

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

(In reply to comment #4)
> > > linux64-ref-platform.build.sjc1.mozilla.com.
> 
> Is this our ref image for Linux64, or just a poorly-named facsimile thereof
> meant to confuse us?

I adjusted the refimage on linux64-ix-ref, so I think option 2.

> > > mw64-ix-slave01.build.mtv1.mozilla.com.
> 
> This is our lone win64 builder AIUI. Should have ping and buildbot
> monitoring AFAICT. Armen?

Hm, I'm about to kill it and send it back to iX as a sub-bug of bug 655304.  I don't see it in production_config.py, so I'm assuming it's not actually in use right now.  I'm also assuming we have a refimage for it.  I guess I should check those assumptions.

Comment 6

[[e-mail] (:bhearsum)]

(In reply to comment #5)
> (In reply to comment #4)
> > > > linux64-ref-platform.build.sjc1.mozilla.com.
> > 
> > Is this our ref image for Linux64, or just a poorly-named facsimile thereof
> > meant to confuse us?
> 
> I adjusted the refimage on linux64-ix-ref, so I think option 2.

(Sorry, meant to reply to this earlier)

That's the 64-bit Linux ref image *VM*, rather than IX.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

OK, and to be clear for Amy:
 linux64-ref-platform - monitor like the other ref platforms
 mw64-ix-slave01 - monitor like the other w64 boxes (which is not at all right now, IIRC?)

Comment 8

[Amy Rich [:arr] [:arich]]

Okay, I think I've added configs for ping for everything suggested in the new nagios configs that will be rolled into place on Monday.

Comment 9

[Amy Rich [:arr] [:arich]]

New checks in place.