650406 - Move monitoring of buildbot-master04/06 from bm-admin01 to admin1

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Amy Rich [:arr] [:arich]]

buildbot-master04 and 06 are in scl1 and should be monitored by admin1, not bm-admin1.

Comment 1

[Amy Rich [:arr] [:arich]]

Many of the hosts we're monitoring with bm-admin01 should be moved to admin1 since they're in scl1.

Comment 3

[Amy Rich [:arr] [:arich]]

admin1.infra.scl1.mozilla.com has address 10.12.75.9

linux64-ix-slave01.build.scl1
linux64-ix-slave02.build.scl1
linux64-ix-slave03.build.scl1
linux64-ix-slave04.build.scl1
linux64-ix-slave05.build.scl1
linux64-ix-slave06.build.scl1
linux64-ix-slave07.build.scl1
linux64-ix-slave08.build.scl1
linux64-ix-slave09.build.scl1
linux64-ix-slave10.build.scl1
linux64-ix-slave11.build.scl1
linux64-ix-slave12.build.scl1
linux64-ix-slave13.build.scl1
linux64-ix-slave14.build.scl1
linux64-ix-slave15.build.scl1
linux64-ix-slave16.build.scl1
linux64-ix-slave17.build.scl1
linux64-ix-slave18.build.scl1
linux64-ix-slave19.build.scl1
linux64-ix-slave20.build.scl1
linux64-ix-slave21.build.scl1
linux64-ix-slave22.build.scl1
linux64-ix-slave23.build.scl1
linux64-ix-slave24.build.scl1
linux64-ix-slave25.build.scl1
linux64-ix-slave26.build.scl1
linux64-ix-slave27.build.scl1
linux64-ix-slave28.build.scl1
linux64-ix-slave29.build.scl1
linux64-ix-slave30.build.scl1
linux64-ix-slave31.build.scl1
linux64-ix-slave32.build.scl1
linux64-ix-slave33.build.scl1
linux64-ix-slave34.build.scl1
linux64-ix-slave35.build.scl1
linux64-ix-slave36.build.scl1
linux64-ix-slave37.build.scl1
linux64-ix-slave38.build.scl1
linux64-ix-slave39.build.scl1
linux64-ix-slave40.build.scl1
linux64-ix-slave41.build.scl1
w32-ix-slave01.build.scl1
w32-ix-slave02.build.scl1
w32-ix-slave03.build.scl1
w32-ix-slave04.build.scl1
w32-ix-slave22.build.scl1
w32-ix-slave23.build.scl1
w32-ix-slave24.build.scl1
w32-ix-slave25.build.scl1
w32-ix-slave26.build.scl1
w32-ix-slave27.build.scl1
w32-ix-slave28.build.scl1
w32-ix-slave29.build.scl1
w32-ix-slave30.build.scl1
w32-ix-slave31.build.scl1
w32-ix-slave32.build.scl1
w32-ix-slave33.build.scl1
w32-ix-slave34.build.scl1
w32-ix-slave35.build.scl1
w32-ix-slave36.build.scl1
w32-ix-slave37.build.scl1
w32-ix-slave38.build.scl1
w32-ix-slave39.build.scl1
w32-ix-slave40.build.scl1
w32-ix-slave41.build.scl1
w32-ix-slave42.build.scl1
linux-ix-slave01.build.scl1
linux-ix-slave02.build.scl1
linux-ix-slave06.build.scl1
linux-ix-slave08.build.scl1
linux-ix-slave12.build.scl1
linux-ix-slave13.build.scl1
linux-ix-slave14.build.scl1
linux-ix-slave16.build.scl1
linux-ix-slave17.build.scl1
linux-ix-slave18.build.scl1
linux-ix-slave19.build.scl1
linux-ix-slave20.build.scl1
linux-ix-slave21.build.scl1
linux-ix-slave22.build.scl1
linux-ix-slave23.build.scl1
linux-ix-slave24.build.scl1
linux-ix-slave25.build.scl1
linux-ix-slave26.build.scl1
linux-ix-slave27.build.scl1
linux-ix-slave28.build.scl1
linux-ix-slave29.build.scl1
linux-ix-slave30.build.scl1
linux-ix-slave31.build.scl1
linux-ix-slave32.build.scl1
linux-ix-slave33.build.scl1
linux-ix-slave34.build.scl1
linux-ix-slave35.build.scl1
linux-ix-slave36.build.scl1
linux-ix-slave37.build.scl1
linux-ix-slave38.build.scl1
linux-ix-slave39.build.scl1
linux-ix-slave40.build.scl1
linux-ix-slave41.build.scl1
linux-ix-slave42.build.scl1
buildbot-master1.build.scl1
buildbot-master2.build.scl1
buildbot-master04.build.scl1
buildbot-master5.build.scl1
buildbot-master06.build.scl1
slavealloc.build.scl1
scl-production-puppet.build.scl1

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: m650406-nrpe-cfg.patch

this should add the necessary hosts to nrpe.  Another patch coming up for NSC.INI

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: m650406-opsi-package-sources-r1.patch

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

buildbot-master1.build.scl1
buildbot-master2.build.scl1
buildbot-master04.build.scl1
buildbot-master5.build.scl1
buildbot-master06.build.scl1
slavealloc.build.scl1
scl-production-puppet.build.scl1

will all need to be updated by hand.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 527297 [details] [diff] [review]
m650406-nrpe-cfg.patch

(patch is reversed, sorry)

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: m650406-opsi-package-sources-r2.patch

version I'm committing (bumps the version number too)

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

OPSI patch works in staging - pushing to production.

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 527297 [details] [diff] [review]
m650406-nrpe-cfg.patch

deployed via patch -R to all of the puppet masters.

Now, this will soak into any active slaves by tomorrow morning, but all of the idle slaves won't get the new data.  However, we can fix that by rebooting those idle slaves tomorrow.  So back to you, Amy.  Don't try to wait out the idle slaves - they stay idle.

Comment 11

[Amy Rich [:arr] [:arich]]

The nagios configs have been changed.  I think Dustin is still battling with OPSI, so I'll pass this over to him for closure when he's done.

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

Enough whacks with the reboot stick and OPSI came around.