873215 - Temporarily migrate SCL3 mac mini racks to 100% A-side power

Status: RESOLVED FIXED

(Infrastructure & Operations :: DCOps, task)

Description

[Derek Moore [:dmoore]]

In order to minimize downtime for the B-side power maintenance in Bug 848086, Release Engineering has requested we preemptively migrate all B-side mac minis to A-side power for the duration of the outage. After maintenance is complete, we can return them to the original configuration.

The following racks will be impacted:

r102-20
r102-21
r102-22

Each rack has 32 mac minis on B-side power. We will migrate each rack one at a time to minimize total impact.

The order of work will be:

1) Install a spare dual-outlet 60A tap can on the A-side power bus
2) Migrate pdu1-1.r102-20 to the new tap can
3) Migrate pdu1-1.r102-21 to the new tap can
4) Move the now-vacant B-side tap can to A-side
5) Migrate pdu1-1.r102-22 to the second tap can

We will reverse this process to return to the original configuration.

Comment 1

[Derek Moore [:dmoore]]

Migration for pdu1-1.r102-20 scheduled for 0900 PDT on 5/17

Comment 2

[Derek Moore [:dmoore]]

s/102/101/g

At :hwine's request, we're going to begin with 101-21 on 5/17

Comment 3

[Amy Rich [:arr] [:arich]]

We've preemptively shut down the two install servers and downtimed them for 2d.  Please make sure they power back on once the work is complete.

Comment 4

[Derek Moore [:dmoore]]

pdu1-1.r101-21 has been migrated to A-side power. Power is restored to all mac minis in r101-21.

Comment 5

[Derek Moore [:dmoore]]

pdu1-1.r101-20 and pdu1-1.r101-22 have been migrated to A-side power. All migrations are complete in preparation for the 6/11 power maintenance in Bug 848086.

Comment 6

[Amy Rich [:arr] [:arich]]

This is now almost a year old.  Did we ever re-balance the power in the racks?

Comment 7

[Van Le [:van]]

>Did we ever re-balance the power in the racks?

We have. It looks like we'd need to create a new MOP if Vantage wants to pursue the maintenance since our infrastructure has changed since then.