527814 - Turn on Nagios monitoring for new talos boxes in MV

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Lukas Blakk [:lsblakk] use ?needinfo]

talos-rev2-leopard16
talos-rev2-linux19
talos-rev2-vista16
talos-rev2-xp23

Comment 1

[Shyam Mani [:fox2mike]]

(In reply to comment #0)
> talos-rev2-leopard16
> talos-rev2-linux19
> talos-rev2-vista16
> talos-rev2-xp23

Okay, so all these machines are in the office it seems and not the San Jose datacenter according to the inventory. I don't see a direct way to add them to Nagios unless we have another Nagios setup in Mountain View.

Comment 2

[Shyam Mani [:fox2mike]]

Might also need DNS/Firewall changes if we want to keep all of build's Nagios checks in one place.

Comment 3

[Lukas Blakk [:lsblakk] use ?needinfo]

> Okay, so all these machines are in the office it seems and not the San Jose
> datacenter according to the inventory. 

That's correct, and also there are 10 more with 'rev1' in their names that will also need nagios monitoring, I believe Phong was looking into that.  There are also at least another 4 - 12 minis that will be coming up in the MV office within the next week.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

From bug 527993: talos, ping check only
qm-pleopard-try12/13/14/15/16 
qm-pubuntu-try12/13/14/15/16

Comment 6

[alice nodelman [:alice] [:anode]]

Need checks for:

talos-rev2-x6401/02/03/04/05/06/07/08/09/10/11/12

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Please add these MV based machines too:
* moz2-darwin9-slave29 thru 37 with same checks as 26
* talos-staging-master02 like talos-staging-master

Comment 8

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

dmoore: 

1) Is this still blocked on MPT DNS?

2) Please add the new talos minis. 
talos-r3-w7-001...020
talos-r3-xp-001...020
talos-r3-leopard001...020

note: While we can test these in staging without nagios, these will need to be nagios monitored before we can use them in production. Hence adding dep.bug.

Comment 9

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #8)
> dmoore: 
> 1) Is this still blocked on MPT DNS?
ping?


> 2) Please add the new talos minis. 
> talos-r3-w7-001...020
> talos-r3-xp-001...020
> talos-r3-leopard001...020
...and also
talos-r3-fed-001...020
talos-r3-fed64-001...020


> note: While we can test these in staging without nagios, these will need to be
> nagios monitored before we can use them in production. Hence adding dep.bug.

Comment 10

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #9)
> (In reply to comment #8)
> > dmoore: 
> > 1) Is this still blocked on MPT DNS?
> ping?

re-ping? According to dmoore, this was no longer blocked on dns, now just machines to be added to nagios. Any ETA?

Comment 11

[Derek Moore [:dmoore]]

All hosts in this bug have been added to the nagios instance on bm-admin01.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #7)
> Please add these MV based machines too:
> * moz2-darwin9-slave29 thru 37 with same checks as 26
> * talos-staging-master02 like talos-staging-master

These ones just have a ping check at the moment - I probably shouldn't have added these machines to a talos bug as they're actually build slaves/masters. Please add the full set of checks using the template machine listed.

Comment 13

[Derek Moore [:dmoore]]

Thanks for the clarification. Nagios updated as requested.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Please add 'root partition' on moz2-darwin9-slave29 thru 37.

Comment 15

[Derek Moore [:dmoore]]

Yeah, sorry I missed that. Fixed.

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Cheers Derek. May this bug remain closed forever more.

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #12)
> (In reply to comment #7)
> > Please add these MV based machines too:
> > * moz2-darwin9-slave29 thru 37 with same checks as 26
> > * talos-staging-master02 like talos-staging-master
> 
> These ones just have a ping check at the moment - I probably shouldn't have
> added these machines to a talos bug as they're actually build slaves/masters.
> Please add the full set of checks using the template machine listed.

(In reply to comment #13)
> Thanks for the clarification. Nagios updated as requested.

This has been reverted somehow. Please reinstate the checks.

Comment 18

[Derek Moore [:dmoore]]

Hmm... not sure what happened here. Rebuilt the nagios configs and everything appears working as it should be.