873239 - move rack 101-21 B side minis to A side May 17

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Hal Wine [:hwine] use NI!]

scripts and record of shutting down minis on B side power of rack 101-21 in scl3

we'll be able to reuse these scripts and procedures when these machines are moved back to B side power.

Overview:
 - T-xxx - relops handles install.test & install.build
 - T-180m - disable buildbot units in slavealloc
 - T-60m - kill any remaining jobs;
         - downtime all units in nagios (including non-buildbot ones)
         - power down buildbot units
 - T-30m - power down mac-signing2
         - power down bld-lion-r5-ref & talos-mtnlion-r5-ref
 - T-0m  - give okay to dcops -- see bug 873215

Comment 1

[Hal Wine [:hwine] use NI!]

Attachment: hosts on B side PDU in rack 101-21

hosts using B side power in rack 101-21 in scl3

Grouped by who:
 - does nagios
   - does powerdown
     - does slavealloc disable

Scripts to come later, I hope.

Comment 2

[Hal Wine [:hwine] use NI!]

Correct host names are found in attachment 750793 [details]:

(In reply to Hal Wine [:hwine] from comment #0)
> Overview:
>  - T-30m - power down mac-signing2
>          - power down bld-lion-r5-ref & talos-mtnlion-r5-ref
           - AND partner-repack1.srv.releng.scl3.mozilla.com

Comment 3

[Armen [:armenzg]]

I have disabled in slavealloc the talos-mtnlion hosts and gracefully shut them down.

Can someone downtime them on nagios please?

Comment 4

[Justin Wood (:Callek)]

(In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) from comment #3)
> I have disabled in slavealloc the talos-mtnlion hosts and gracefully shut
> them down.
> 
> Can someone downtime them on nagios please?

All mtnlion hosts are downtimed for 5 hours (4.5 actually by now)

Comment 5

[Hal Wine [:hwine] use NI!]

downtimed the non-mtnlion hosts:
    mac-signing2.srv.releng.scl3.mozilla.com
    bld-lion-r5-ref.build.releng.scl3.mozilla.com
    partner-repack1.srv.releng.scl3.mozilla.com

Comment 6

[[e-mail] (:bhearsum)]

mac-signing2 is offline.

Comment 7

[Hal Wine [:hwine] use NI!]

powered down:
    bld-lion-r5-ref.build.releng.scl3.mozilla.com
    partner-repack1.srv.releng.scl3.mozilla.com

Comment 8

[Hal Wine [:hwine] use NI!]

powered down buildbot hosts

Comment 9

[Armen [:armenzg]]

The buildbot hosts are backed to the pool.
The other hosts are as well up and running.

DCOps, anything left on your side?

Comment 10

[Hal Wine [:hwine] use NI!]

Attachment: fabric file to shutdown all affected buildbot controlled hosts

For use when we switch back.

With fabric module in path, use:
 fab -f fabfile-bz873239.py -j shut_down

Comment 11

[Hal Wine [:hwine] use NI!]

Work to move to A side completed. Will keep bug open until we migrate back after the main DC work is done.

Comment 12

[Hal Wine [:hwine] use NI!]

move back can be scheduled after bug 848086 closed

Comment 13

[Hal Wine [:hwine] use NI!]

Closing bug to reduce confusion with other power moves.