Closed Bug 920667 Opened 10 years ago Closed 10 years ago

New host group with in Nagios

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Platform:
x86_64
Windows 7
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: markco, Assigned: ashish)

References

Details

We are going to be transitioning to different OS build for Windows 2008R2. However, this transition will be incremental. I was wondering if we could create an addition host group to windows 2008R2 64 bit build hosts (w64-ix-slaves) that will not monitor the E disk? 

If the host group can be create could the following machines be added:

w64-ix-slave110
w64-ix-slave111
w64-ix-slave112
w64-ix-slave113
w64-ix-slave114
w64-ix-slave115
w64-ix-slave116
w64-ix-slave117
w64-ix-slave118
w64-ix-slave119

Thank you.
Blocks: 781277
w64-ix-slave10 will need to be added to this list.
I'm moving this bug to the IT triage queue (product mozilla.org, component Server Operations) so that the appropriate team can take action.

I apologize for the one week delay in recognizing that it was misfiled; this happens a few times a week and once in a while a bug is missed in the noise.

Please use the IT triage queue when filing these requests - I understand it's virtually impossible to discover organically, but it ensures that requests are handled in a timely manner.

'mozilla.org :: Server Operations' works in the enter_bug.cgi quicksearch, and the exact match should be first in the list.
Assignee: infra → server-ops
Component: Infrastructure: Monitoring → Server Operations
Product: Infrastructure & Operations → mozilla.org
QA Contact: jdow → shyam
For the lack of a more descriptive name, I chose w64r2-ix-slaves but would really like something more obvious. FWIW, monitoring on E: for these machines have been removed.

Please do drop in with a better name :)
Status: NEW → RESOLVED
Closed: 10 years ago
Flags: needinfo?(mcornmesser)
Resolution: --- → FIXED
As far as the name, I will put some thought into the name. For more obvious are we just talking about to differentiate between the two builds or additional data in the name? 

Also could you add these machines to the group as well: 

w64-ix-slave120 
w64-ix-slave121 
w64-ix-slave122 
w64-ix-slave123 
w64-ix-slave124 
w64-ix-slave125 
w64-ix-slave126 
w64-ix-slave127 
w64-ix-slave128 
w64-ix-slave129 
w64-ix-slave130

This is going to be an ongoing transition. Should we open additional bugs for each group, or should we keep this one bug open until the transition is complete?
Flags: needinfo?(ashish)
(In reply to Mark Cornmesser [:markco] from comment #4)
> As far as the name, I will put some thought into the name. For more obvious
> are we just talking about to differentiate between the two builds or
> additional data in the name? 
> 
Oh this is just in Nagios. I do not touch any other part of the system for releng/ops.

> Also could you add these machines to the group as well: 
> 
> w64-ix-slave120 
> w64-ix-slave121 
> w64-ix-slave122 
> w64-ix-slave123 
> w64-ix-slave124 
> w64-ix-slave125 
> w64-ix-slave126 
> w64-ix-slave127 
> w64-ix-slave128 
> w64-ix-slave129 
> w64-ix-slave130
>
Done
 
> This is going to be an ongoing transition. Should we open additional bugs
> for each group, or should we keep this one bug open until the transition is
> complete?
>
Let's reuse this bug.
Status: RESOLVED → REOPENED
Flags: needinfo?(ashish)
Resolution: FIXED → ---
ashish: I'm not sure theres a more obvious name to be had.  Once we finish converting them all, you can go back to calling them w64-ix if you like (and just remove the E: check form the old group, change the all the hosts back to the old group, and delete the new group).  Having two sets is a temporary state while we reimage batches of them.
Flags: needinfo?(mcornmesser)
:arr no problem, I'm not terribly particular especially if this is all temporary.
Assignee: server-ops → ashish
Status: REOPENED → ASSIGNED
Can you move w64-ix-slave30 to the w64r2-ix-slaves host group as well, please?
Flags: needinfo?(ashish)
Done!
Flags: needinfo?(ashish)
Please move these to the w64r2-ix-slaves host group:

w64-ix-slave06
w64-ix-slave07
w64-ix-slave08
w64-ix-slave09
w64-ix-slave10
w64-ix-slave30
w64-ix-slave31
w64-ix-slave32
w64-ix-slave33
w64-ix-slave34
as well as these:

w64-ix-slave11
w64-ix-slave12
w64-ix-slave13
w64-ix-slave14
w64-ix-slave15
w64-ix-slave35
w64-ix-slave36
w64-ix-slave37
w64-ix-slave38
w64-ix-slave39
Flags: needinfo?(ashish)
Done!
Flags: needinfo?(ashish)
Please move these to the w64r2-ix-slaves host group:

w64-ix-slave16
w64-ix-slave17
w64-ix-slave18
w64-ix-slave19
w64-ix-slave20
w64-ix-slave40
w64-ix-slave41
w64-ix-slave44
w64-ix-slave45
w64-ix-slave46
as well as these:

w64-ix-slave21
w64-ix-slave23
w64-ix-slave24
w64-ix-slave25
w64-ix-slave26
w64-ix-slave47
w64-ix-slave48
w64-ix-slave49
w64-ix-slave50
w64-ix-slave51
Flags: needinfo?(ashish)
Done!
Flags: needinfo?(ashish)
Please move these to the w64r2-ix-slaves host group:

w64-ix-slave27
w64-ix-slave28
w64-ix-slave29
w64-ix-slave42
w64-ix-slave43
w64-ix-slave52
w64-ix-slave53
w64-ix-slave54
w64-ix-slave55
w64-ix-slave56
Flags: needinfo?(ashish)
as well as these:

w64-ix-slave75
w64-ix-slave76
w64-ix-slave77
w64-ix-slave78
w64-ix-slave79
w64-ix-slave57
w64-ix-slave58
w64-ix-slave59
w64-ix-slave60
w64-ix-slave61
Done!
Flags: needinfo?(ashish)
Please move these to the w64r2-ix-slaves host group:

w64-ix-slave81
w64-ix-slave82
w64-ix-slave83
w64-ix-slave84
w64-ix-slave87
w64-ix-slave62
w64-ix-slave63
w64-ix-slave64
w64-ix-slave65
w64-ix-slave66
Flags: needinfo?(ashish)
these too:

w64-ix-slave88
w64-ix-slave89
w64-ix-slave90
w64-ix-slave91
w64-ix-slave92
w64-ix-slave67
w64-ix-slave68
w64-ix-slave69
w64-ix-slave70
w64-ix-slave71
Done!
Flags: needinfo?(ashish)
Please move these to the w64r2-ix-slaves host group:

w64-ix-slave72
w64-ix-slave73
w64-ix-slave74
Flags: needinfo?(ashish)
Done!
Flags: needinfo?(ashish)
Please move these to the w64r2-ix-slaves host group:

w64-ix-slave04
w64-ix-slave22
w64-ix-slave80
Flags: needinfo?(ashish)
(In reply to John Hopkins (:jhopkins) from comment #24)
> Please move these to the w64r2-ix-slaves host group:
> 
> w64-ix-slave04
> w64-ix-slave22
> w64-ix-slave80
-

Just saw this come through from watching the component.  Figured I would knock it out.
Moved these 3 hosts to w64r2-ix-slaves Hostgroup.
Flags: needinfo?(ashish)
Please move this to the w64r2-ix-slaves host group:

w64-ix-slave03
These as well:

w64-ix-slave93
w64-ix-slave94
w64-ix-slave95
w64-ix-slave96
w64-ix-slave97
Flags: needinfo?(ashish)
Done!
Flags: needinfo?(ashish)
Please move these to the w64r2-ix-slaves host group:

w64-ix-slave98
w64-ix-slave99
w64-ix-slave100
w64-ix-slave101
w64-ix-slave102
Flags: needinfo?(ashish)
Also these:

w64-ix-slave103
w64-ix-slave104
w64-ix-slave105
w64-ix-slave106
w64-ix-slave107
Done!
Flags: needinfo?(ashish)
Please move these to the w64r2-ix-slaves host group:

w64-ix-slave108
w64-ix-slave109
w64-ix-slave131
w64-ix-slave132
w64-ix-slave133
Flags: needinfo?(ashish)
Done!
Flags: needinfo?(ashish)
At this point, I think we should move all of the hosts to one unified group that does not have a check for e: (we never ever see alerts for e: anyway) and go back to calling it by the original name again.
Blocks: 918414
No longer blocks: 781277
I'd slightly prefer to keep drive e: monitoring active on rev1's to keep murphy's law from biting us.  But if it greatly simplifies the nagios configs, probably worth the tradeoff since e: rarely/never alerts.

Please move these to the w64r2-ix-slaves host group:

w64-ix-slave134
w64-ix-slave135
w64-ix-slave136
w64-ix-slave137
w64-ix-slave138
Flags: needinfo?(ashish)
Alright I'm gonna keep things as they are. slave[134-138] added to w64r2-ix-slaves.
Flags: needinfo?(ashish)
Hi Ashish, can you please add the new rev2 nagios checks for these machines? Currently they have the old Win64 rev1 checks.
Should we change the default nagios checks for Win64 machines to be the ones for rev2 machines?

Thanks for your help!

w64-ix-slave159.build.scl1.mozilla.com
w64-ix-slave160.build.scl1.mozilla.com
w64-ix-slave161.build.scl1.mozilla.com
w64-ix-slave162.build.scl1.mozilla.com
w64-ix-slave163.build.scl1.mozilla.com
w64-ix-slave164.build.scl1.mozilla.com
w64-ix-slave165.build.scl1.mozilla.com
w64-ix-slave166.build.scl1.mozilla.com
w64-ix-slave167.build.scl1.mozilla.com
w64-ix-slave168.build.scl1.mozilla.com
w64-ix-slave169.build.scl1.mozilla.com
w64-ix-slave170.build.scl1.mozilla.com
Flags: needinfo?(ashish)
w64-ix-slave[159-170] added to w64r2-ix-slaves hostgroup.

:armenzg There are still a handful of slaves that are still in the old hostgroup, w64-ix-slave[139-157] for instance. rev2 in Nagios only removes the E: check and yes, that can be set as default for all. I'm on PTO for the reminder of this week and can make significant changes on Monday (Sunday evening Pacific). However, others in SRE should be able to add hosts to w64r2-ix-slaves on short notice.
Flags: needinfo?(ashish)
Please move these to the w64r2-ix-slaves host group:

w64-ix-slave139
w64-ix-slave140
w64-ix-slave141
w64-ix-slave142
w64-ix-slave143
Flags: needinfo?(ashish)
Done!
Flags: needinfo?(ashish)
Amy said earlier: At this point, I think we should move all of the hosts to one unified group that does not have a check for e: (we never ever see alerts for e: anyway) and go back to calling it by the original name again.

I'm reimaging the last of the rev1 w64-ix-slaves so I think we should go ahead with that plan whenever is convenient.
One last request in comment 40, Ashish.  Thanks!
Flags: needinfo?(ashish)
I'll clean this up and remove the w64r2-ix-slaves hostgroup and exclude the E: disk check
Completed in sysadmins r84631.

I've retained the w64-ix-slaves hostgroup and removed w64r2-ix-slaves and the E: disk check.
Status: ASSIGNED → RESOLVED
Closed: 10 years ago → 10 years ago
Flags: needinfo?(ashish)
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.