Closed Bug 1020424 Opened 10 years ago Closed 10 years ago

nagios monitoring for new releng seamicro windows nodes and modifications to existing windows monitoring

Categories

(mozilla.org Graveyard :: Server Operations, task)

x86
macOS
task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: arich, Assigned: ashish)

References

Details

Attachments

(1 file)

As part of bug 1014703 we're working on standing up 64 new windows machines on seamicro-c1.r101-3.console.scl3.mozilla.com (which, according to ashish should be the parent for these new hosts).  We're also in the process of moving machines from scl1 to scl3 and splitting out the winbuild and wintry vlans.

I'd like to modify the existing nagios monitoring config so that we clean up the config and normalize it to reflect the groupings we care about.  Please feel free to ping me if any of this is unclear or needs further discussion.

1) have two different groups for wintry and winbuild so we can have separate cluster checks for each of them (aka some machines that are currently in the same group as the builders will be split out to a new try group). The checks and thresholds should remain the same as the existing group for the builders.

Right now it looks like we have the following overlapping references for w64 (in scl3) and b2008 that we should consolidate:

services.pp:
w64-ix-slaves and b2008-ix-slaves RENAME to 2008-build-slaves (will be a mix of ix and sm)
CREATE 2008-try-slaves and use it for all b-2008*.wintry.releng.scl3.mozilla.com hosts

hostgroups.pp:
w64-ix-slaves and b2008-ix-slaves RENAME to 2008-build-slaves (will be a mix of ix and sm)
CREATE 2008-try-slaves and use it for all b-2008*.wintry.releng.scl3.mozilla.com hosts

scl3.pp:
scl3-b2008-ix-ping and scl3-w64-slave-ping RENAME to scl3-2008-build-slave-ping (will be a mix of ix and sm)
CREATE scl3-2008-try-slave-ping and use it for all b-2008*.wintry.releng.scl3.mozilla.com hosts
b2008-ix-slaves and w64-ix-slaves RENAME to 2008-build-slaves (will be a mix of ix and sm)
CREATE 2008-try-slaves and use it for all b-2008*.wintry.releng.scl3.mozilla.com hosts

clusterchecks.pp:
scl3-w64-slave-ping and scl3-b2008-ix-ping RENAME to scl3-2008-build-slave-ping (will be a mix of ix and sm)
CREATE scl3-2008-try-slave-ping and use it for all b-2008*.wintry.releng.scl3.mozilla.com hosts

2) Make sure that we follow the convention for additional hosts that we move from scl1 -> scl3.

3) add the machines below to the appropriate groups and downtime them for a week:

scl3-2008-try-slave-ping and 2008-try-slaves:

b-2008-sm-0001.wintry.releng.scl3.mozilla.com
b-2008-sm-0002.wintry.releng.scl3.mozilla.com
b-2008-sm-0003.wintry.releng.scl3.mozilla.com
b-2008-sm-0004.wintry.releng.scl3.mozilla.com
b-2008-sm-0005.wintry.releng.scl3.mozilla.com
b-2008-sm-0006.wintry.releng.scl3.mozilla.com
b-2008-sm-0007.wintry.releng.scl3.mozilla.com
b-2008-sm-0080.wintry.releng.scl3.mozilla.com
b-2008-sm-0009.wintry.releng.scl3.mozilla.com
b-2008-sm-0010.wintry.releng.scl3.mozilla.com
b-2008-sm-0011.wintry.releng.scl3.mozilla.com
b-2008-sm-0012.wintry.releng.scl3.mozilla.com
b-2008-sm-0013.wintry.releng.scl3.mozilla.com
b-2008-sm-0014.wintry.releng.scl3.mozilla.com
b-2008-sm-0015.wintry.releng.scl3.mozilla.com
b-2008-sm-0016.wintry.releng.scl3.mozilla.com
b-2008-sm-0017.wintry.releng.scl3.mozilla.com
b-2008-sm-0018.wintry.releng.scl3.mozilla.com
b-2008-sm-0019.wintry.releng.scl3.mozilla.com
b-2008-sm-0020.wintry.releng.scl3.mozilla.com
b-2008-sm-0021.wintry.releng.scl3.mozilla.com
b-2008-sm-0022.wintry.releng.scl3.mozilla.com
b-2008-sm-0023.wintry.releng.scl3.mozilla.com
b-2008-sm-0024.wintry.releng.scl3.mozilla.com
b-2008-sm-0025.wintry.releng.scl3.mozilla.com
b-2008-sm-0026.wintry.releng.scl3.mozilla.com
b-2008-sm-0027.wintry.releng.scl3.mozilla.com
b-2008-sm-0028.wintry.releng.scl3.mozilla.com
b-2008-sm-0029.wintry.releng.scl3.mozilla.com
b-2008-sm-0030.wintry.releng.scl3.mozilla.com
b-2008-sm-0031.wintry.releng.scl3.mozilla.com
b-2008-sm-0032.wintry.releng.scl3.mozilla.com

scl3-2008-build-slave-ping and 2008-build-slaves:

b-2008-sm-0033.winbuild.releng.scl3.mozilla.com
b-2008-sm-0034.winbuild.releng.scl3.mozilla.com
b-2008-sm-0035.winbuild.releng.scl3.mozilla.com
b-2008-sm-0036.winbuild.releng.scl3.mozilla.com
b-2008-sm-0037.winbuild.releng.scl3.mozilla.com
b-2008-sm-0038.winbuild.releng.scl3.mozilla.com
b-2008-sm-0039.winbuild.releng.scl3.mozilla.com
b-2008-sm-0040.winbuild.releng.scl3.mozilla.com
b-2008-sm-0041.winbuild.releng.scl3.mozilla.com
b-2008-sm-0042.winbuild.releng.scl3.mozilla.com
b-2008-sm-0043.winbuild.releng.scl3.mozilla.com
b-2008-sm-0044.winbuild.releng.scl3.mozilla.com
b-2008-sm-0045.winbuild.releng.scl3.mozilla.com
b-2008-sm-0046.winbuild.releng.scl3.mozilla.com
b-2008-sm-0047.winbuild.releng.scl3.mozilla.com
b-2008-sm-0048.winbuild.releng.scl3.mozilla.com
b-2008-sm-0049.winbuild.releng.scl3.mozilla.com
b-2008-sm-0050.winbuild.releng.scl3.mozilla.com
b-2008-sm-0051.winbuild.releng.scl3.mozilla.com
b-2008-sm-0052.winbuild.releng.scl3.mozilla.com
b-2008-sm-0053.winbuild.releng.scl3.mozilla.com
b-2008-sm-0054.winbuild.releng.scl3.mozilla.com
b-2008-sm-0055.winbuild.releng.scl3.mozilla.com
b-2008-sm-0056.winbuild.releng.scl3.mozilla.com
b-2008-sm-0057.winbuild.releng.scl3.mozilla.com
b-2008-sm-0058.winbuild.releng.scl3.mozilla.com
b-2008-sm-0059.winbuild.releng.scl3.mozilla.com
b-2008-sm-0060.winbuild.releng.scl3.mozilla.com
b-2008-sm-0061.winbuild.releng.scl3.mozilla.com
b-2008-sm-0062.winbuild.releng.scl3.mozilla.com
b-2008-sm-0063.winbuild.releng.scl3.mozilla.com
b-2008-sm-0064.winbuild.releng.scl3.mozilla.com
Uh, b-2008-sm-0080.wintry.releng.scl3.mozilla.com should be b-2008-sm-0008.wintry.releng.scl3.mozilla.com.
The reorg of the existing nodes can wait till after the move trains (as discussed with dmoore), but can we get the new seamicro nodes set up this week, please?
Ok I'll work on this today
Assignee: server-ops → ashish
Status: NEW → ASSIGNED
Merged w64-ix-slaves and b2008-ix-slaves hostgroups into 2008-build-slaves hostgroup in sysadmins r88888 (nice rev no.!)
2008-try-slaves added in sysadmins r88942.
2008-build-slaves added in syadmins 88945.
Merged scl3-w64-slave-ping and scl3-b2008-ix-ping servicegroups into scl3-2008-build-slave-ping servicegroup in sysadmins r88947.
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Attached file diff
It looks like the scl3-2008-try-slave-ping and scl3-2008-build-slave-ping servicegroups modifications for the existing machines in scl3 were missed.  I went back and modified those.
And the existing hostgroups were missed as well, so I also updated those.
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: