Closed Bug 897111 Opened 12 years ago Closed 12 years ago

Power maintenance in SCL3 between 7/24 and 8/2

Categories

(Infrastructure & Operations :: DCOps, task)

x86
macOS
task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: dmoore, Unassigned)

References

Details

For the final phase of the SCL3 expansion, we will need to de-energize both the primary (A) and secondary (B) power feeds in order to expand the power bus. We have refined this process during previous maintenance (example: Bug 848086). The current proposed schedule is as follows: 7/24 - B-side power is cut from 0500 to 1700 PDT 7/25 - B-side power is cut from 0500 to 1700 PDT 7/26 - No work 7/27 - No work 7/28 - No work 7/29 - 1/3 of the mac mini pool is migrated to B-side power from 0800 to 1200 PDT 7/30 - 1/3 of the mac mini pool is migrated to B-side power from 0600 to 1000 PDT 7/31 - 1/3 of the mac mini pool is migrated to B-side power from 0600 to 1000 PDT 8/01 - A-side power is cut from 0500 to 1700 PDT 8/02 - A-side power is cut from 0500 to 1700 PDT, work is complete Due to prior scheduled maintenance, all single-corded equipment has already been migrated to A-side power. None of this equipment will be impacted by the work on 7/24 and 7/25. DCOps will then begin working with release engineering to migrate all single-corded mac minis to B-side power prior to 8/1.
colo-trip: --- → scl3
During prior maintenance, we encountered the follow issues: 1) SeaMicro chassis ATS were not 100% reliable. We will be manually failing over to the appropriate power feeds vs. relying on automatic transfer. 2) OOB switches are single-corded. It was inconvenient to operate for extended periods without OOB access, so we will be restoring OOB access after a brief outage. 3) The 'oobcore1' switch did not have redundant power connected. This was corrected.
We're going to bump up the 7/29 mac mini migration start time. Current schedule is as follows: 7/24 - B-side power is cut from 0500 to 1700 PDT 7/25 - B-side power is cut from 0500 to 1700 PDT 7/26 - No work 7/27 - No work 7/28 - No work 7/29 - 1/3 of the mac mini pool is migrated to B-side power from 0500 to 0900 PDT 7/30 - 1/3 of the mac mini pool is migrated to B-side power from 0600 to 1000 PDT 7/31 - 1/3 of the mac mini pool is migrated to B-side power from 0600 to 1000 PDT 8/01 - A-side power is cut from 0500 to 1700 PDT 8/02 - A-side power is cut from 0500 to 1700 PDT, work is complete
Depends on: 897241
Depends on: 897244
Depends on: 897246
This work has been cancelled by the facility at the request of a neighboring tenant. We will not be allowed to proceed with power maintenance on 7/24 or 7/25, at the very least. We are regrouping tomorrow with all involved parties in order to create an alternate scheduling proposal. I will provide another update at that time.
It took a few days to get consensus from the other tenants, but we now have our next schedule proposal: 7/29 - B-side power is cut at 0500 PDT 7/30 - B-side power is restored at 1700 PDT 7/31 - 1/3 of the mac mini pool is migrated to B-side power from 0600 to 1000 PDT 8/01 - 1/3 of the mac mini pool is migrated to B-side power from 0600 to 1000 PDT 8/02 - 1/3 of the mac mini pool is migrated to B-side power from 0600 to 1000 PDT 8/03 - No work 8/04 - No work 8/05 - A-side power is cut at 0500 PDT 8/06 - A-side power is restored at 1700 PDT I'll be reaching out to the impacted groups directly in order to confirm Mozilla approval of this schedule.
This schedule was approved by all tenants and is proceeding as planned.
B-side power is fully de-energized and will be restored within 48 hours.
The B-side electrical installation and expansion work has been approved by the city inspector. We are still on track to restore power by the 48-hour mark.
B-side power is restored at the 48-hour mark. We are now on fully-redundant power until A-side power is de-energized at 0500 PDT on 8/5.
All mac mini's have been moved to B-side. ready for A-side power work from releng side.
This power work has been completed, as scheduled.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.