875655 - move rack 101-22 B side minis to A side May 24

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Hal Wine [:hwine] use NI!]

+++ This bug was initially created as a clone of Bug #873239 +++

scripts and record of shutting down minis on B side power of rack 101-21 in scl3

we'll be able to reuse these scripts and procedures when these machines are moved back to B side power.

Overview:
 - T-xxx - relops handles install.test & install.build
 - T-180m - disable buildbot units in slavealloc
 - T-60m - kill any remaining jobs;
         - downtime all units in nagios (including non-buildbot ones)
         - power down buildbot units
 - T-0m  - give okay to dcops -- see bug 873215

Comment 1

[Hal Wine [:hwine] use NI!]

Attachment: Units on B side of rack 101-22

List of minis to be powered down

Comment 2

[Hal Wine [:hwine] use NI!]

Attachment: fabric file to shutdown all affected buildbot controlled hosts

To power down the affected machines

Comment 3

[Hal Wine [:hwine] use NI!]

Attachment: record the current enabled state of B side machines

Comment 4

[Hal Wine [:hwine] use NI!]

Attachment: disable all B side slaves

Comment 5

[Hal Wine [:hwine] use NI!]

recorded existing enable state using attachment 753639 [details]. Two hosts currently disabled: bld-lion-r5-052 & bld-lion-r5-054

disabled b side slaves using attachment 753641 [details]

Comment 6

[Van Le [:van]]

worked with :hwine to move 101-22 B power over to A side.

Comment 7

[Hal Wine [:hwine] use NI!]

machines re-enabled in slavealloc and rebooted (thanks to :van for help) to get back in service.

Work completed for now, move back can be scheduled after bug 848086 scheduled.

Comment 8

[Hal Wine [:hwine] use NI!]

Attachment: fabric file to shutdown all affected buildbot controlled hosts

added "reboot" and "verify" steps

Comment 9

[Hal Wine [:hwine] use NI!]

Attachment: fabric file to shutdown all affected buildbot controlled hosts

use safer 'reboot' as cltbld procedure, can also be run in parallel.

Comment 10

[Hal Wine [:hwine] use NI!]

Attachment: record the current enabled state of B side machines

correct to use short hostname

Comment 11

[Hal Wine [:hwine] use NI!]

Closing bug to reduce confusion with other power moves.