nagios monitoring for new releng seamicro windows nodes and modifications to existing windows monitoring

RESOLVED FIXED

Status

mozilla.org Graveyard
Server Operations
RESOLVED FIXED
4 years ago
3 years ago

People

(Reporter: arr, Assigned: ashish)

Tracking

Details

Attachments

(1 attachment)

(Reporter)

Description

4 years ago
As part of bug 1014703 we're working on standing up 64 new windows machines on seamicro-c1.r101-3.console.scl3.mozilla.com (which, according to ashish should be the parent for these new hosts).  We're also in the process of moving machines from scl1 to scl3 and splitting out the winbuild and wintry vlans.

I'd like to modify the existing nagios monitoring config so that we clean up the config and normalize it to reflect the groupings we care about.  Please feel free to ping me if any of this is unclear or needs further discussion.

1) have two different groups for wintry and winbuild so we can have separate cluster checks for each of them (aka some machines that are currently in the same group as the builders will be split out to a new try group). The checks and thresholds should remain the same as the existing group for the builders.

Right now it looks like we have the following overlapping references for w64 (in scl3) and b2008 that we should consolidate:

services.pp:
w64-ix-slaves and b2008-ix-slaves RENAME to 2008-build-slaves (will be a mix of ix and sm)
CREATE 2008-try-slaves and use it for all b-2008*.wintry.releng.scl3.mozilla.com hosts

hostgroups.pp:
w64-ix-slaves and b2008-ix-slaves RENAME to 2008-build-slaves (will be a mix of ix and sm)
CREATE 2008-try-slaves and use it for all b-2008*.wintry.releng.scl3.mozilla.com hosts

scl3.pp:
scl3-b2008-ix-ping and scl3-w64-slave-ping RENAME to scl3-2008-build-slave-ping (will be a mix of ix and sm)
CREATE scl3-2008-try-slave-ping and use it for all b-2008*.wintry.releng.scl3.mozilla.com hosts
b2008-ix-slaves and w64-ix-slaves RENAME to 2008-build-slaves (will be a mix of ix and sm)
CREATE 2008-try-slaves and use it for all b-2008*.wintry.releng.scl3.mozilla.com hosts

clusterchecks.pp:
scl3-w64-slave-ping and scl3-b2008-ix-ping RENAME to scl3-2008-build-slave-ping (will be a mix of ix and sm)
CREATE scl3-2008-try-slave-ping and use it for all b-2008*.wintry.releng.scl3.mozilla.com hosts

2) Make sure that we follow the convention for additional hosts that we move from scl1 -> scl3.

3) add the machines below to the appropriate groups and downtime them for a week:

scl3-2008-try-slave-ping and 2008-try-slaves:

b-2008-sm-0001.wintry.releng.scl3.mozilla.com
b-2008-sm-0002.wintry.releng.scl3.mozilla.com
b-2008-sm-0003.wintry.releng.scl3.mozilla.com
b-2008-sm-0004.wintry.releng.scl3.mozilla.com
b-2008-sm-0005.wintry.releng.scl3.mozilla.com
b-2008-sm-0006.wintry.releng.scl3.mozilla.com
b-2008-sm-0007.wintry.releng.scl3.mozilla.com
b-2008-sm-0080.wintry.releng.scl3.mozilla.com
b-2008-sm-0009.wintry.releng.scl3.mozilla.com
b-2008-sm-0010.wintry.releng.scl3.mozilla.com
b-2008-sm-0011.wintry.releng.scl3.mozilla.com
b-2008-sm-0012.wintry.releng.scl3.mozilla.com
b-2008-sm-0013.wintry.releng.scl3.mozilla.com
b-2008-sm-0014.wintry.releng.scl3.mozilla.com
b-2008-sm-0015.wintry.releng.scl3.mozilla.com
b-2008-sm-0016.wintry.releng.scl3.mozilla.com
b-2008-sm-0017.wintry.releng.scl3.mozilla.com
b-2008-sm-0018.wintry.releng.scl3.mozilla.com
b-2008-sm-0019.wintry.releng.scl3.mozilla.com
b-2008-sm-0020.wintry.releng.scl3.mozilla.com
b-2008-sm-0021.wintry.releng.scl3.mozilla.com
b-2008-sm-0022.wintry.releng.scl3.mozilla.com
b-2008-sm-0023.wintry.releng.scl3.mozilla.com
b-2008-sm-0024.wintry.releng.scl3.mozilla.com
b-2008-sm-0025.wintry.releng.scl3.mozilla.com
b-2008-sm-0026.wintry.releng.scl3.mozilla.com
b-2008-sm-0027.wintry.releng.scl3.mozilla.com
b-2008-sm-0028.wintry.releng.scl3.mozilla.com
b-2008-sm-0029.wintry.releng.scl3.mozilla.com
b-2008-sm-0030.wintry.releng.scl3.mozilla.com
b-2008-sm-0031.wintry.releng.scl3.mozilla.com
b-2008-sm-0032.wintry.releng.scl3.mozilla.com

scl3-2008-build-slave-ping and 2008-build-slaves:

b-2008-sm-0033.winbuild.releng.scl3.mozilla.com
b-2008-sm-0034.winbuild.releng.scl3.mozilla.com
b-2008-sm-0035.winbuild.releng.scl3.mozilla.com
b-2008-sm-0036.winbuild.releng.scl3.mozilla.com
b-2008-sm-0037.winbuild.releng.scl3.mozilla.com
b-2008-sm-0038.winbuild.releng.scl3.mozilla.com
b-2008-sm-0039.winbuild.releng.scl3.mozilla.com
b-2008-sm-0040.winbuild.releng.scl3.mozilla.com
b-2008-sm-0041.winbuild.releng.scl3.mozilla.com
b-2008-sm-0042.winbuild.releng.scl3.mozilla.com
b-2008-sm-0043.winbuild.releng.scl3.mozilla.com
b-2008-sm-0044.winbuild.releng.scl3.mozilla.com
b-2008-sm-0045.winbuild.releng.scl3.mozilla.com
b-2008-sm-0046.winbuild.releng.scl3.mozilla.com
b-2008-sm-0047.winbuild.releng.scl3.mozilla.com
b-2008-sm-0048.winbuild.releng.scl3.mozilla.com
b-2008-sm-0049.winbuild.releng.scl3.mozilla.com
b-2008-sm-0050.winbuild.releng.scl3.mozilla.com
b-2008-sm-0051.winbuild.releng.scl3.mozilla.com
b-2008-sm-0052.winbuild.releng.scl3.mozilla.com
b-2008-sm-0053.winbuild.releng.scl3.mozilla.com
b-2008-sm-0054.winbuild.releng.scl3.mozilla.com
b-2008-sm-0055.winbuild.releng.scl3.mozilla.com
b-2008-sm-0056.winbuild.releng.scl3.mozilla.com
b-2008-sm-0057.winbuild.releng.scl3.mozilla.com
b-2008-sm-0058.winbuild.releng.scl3.mozilla.com
b-2008-sm-0059.winbuild.releng.scl3.mozilla.com
b-2008-sm-0060.winbuild.releng.scl3.mozilla.com
b-2008-sm-0061.winbuild.releng.scl3.mozilla.com
b-2008-sm-0062.winbuild.releng.scl3.mozilla.com
b-2008-sm-0063.winbuild.releng.scl3.mozilla.com
b-2008-sm-0064.winbuild.releng.scl3.mozilla.com
(Reporter)

Comment 1

4 years ago
Uh, b-2008-sm-0080.wintry.releng.scl3.mozilla.com should be b-2008-sm-0008.wintry.releng.scl3.mozilla.com.
(Reporter)

Comment 2

4 years ago
The reorg of the existing nodes can wait till after the move trains (as discussed with dmoore), but can we get the new seamicro nodes set up this week, please?
(Assignee)

Comment 3

4 years ago
Ok I'll work on this today
Assignee: server-ops → ashish
Status: NEW → ASSIGNED
(Assignee)

Comment 4

4 years ago
Merged w64-ix-slaves and b2008-ix-slaves hostgroups into 2008-build-slaves hostgroup in sysadmins r88888 (nice rev no.!)
(Assignee)

Comment 5

4 years ago
2008-try-slaves added in sysadmins r88942.
(Assignee)

Comment 6

4 years ago
2008-build-slaves added in syadmins 88945.
(Assignee)

Comment 7

4 years ago
Merged scl3-w64-slave-ping and scl3-b2008-ix-ping servicegroups into scl3-2008-build-slave-ping servicegroup in sysadmins r88947.
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
(Reporter)

Comment 8

4 years ago
Created attachment 8438682 [details]
diff

It looks like the scl3-2008-try-slave-ping and scl3-2008-build-slave-ping servicegroups modifications for the existing machines in scl3 were missed.  I went back and modified those.
(Reporter)

Comment 9

4 years ago
And the existing hostgroups were missed as well, so I also updated those.
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.