Closed
Bug 897111
Opened 12 years ago
Closed 12 years ago
Power maintenance in SCL3 between 7/24 and 8/2
Categories
(Infrastructure & Operations :: DCOps, task)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: dmoore, Unassigned)
References
Details
For the final phase of the SCL3 expansion, we will need to de-energize both the primary (A) and secondary (B) power feeds in order to expand the power bus.
We have refined this process during previous maintenance (example: Bug 848086).
The current proposed schedule is as follows:
7/24 - B-side power is cut from 0500 to 1700 PDT
7/25 - B-side power is cut from 0500 to 1700 PDT
7/26 - No work
7/27 - No work
7/28 - No work
7/29 - 1/3 of the mac mini pool is migrated to B-side power from 0800 to 1200 PDT
7/30 - 1/3 of the mac mini pool is migrated to B-side power from 0600 to 1000 PDT
7/31 - 1/3 of the mac mini pool is migrated to B-side power from 0600 to 1000 PDT
8/01 - A-side power is cut from 0500 to 1700 PDT
8/02 - A-side power is cut from 0500 to 1700 PDT, work is complete
Due to prior scheduled maintenance, all single-corded equipment has already been migrated to A-side power. None of this equipment will be impacted by the work on 7/24 and 7/25. DCOps will then begin working with release engineering to migrate all single-corded mac minis to B-side power prior to 8/1.
Reporter | ||
Updated•12 years ago
|
colo-trip: --- → scl3
Reporter | ||
Comment 1•12 years ago
|
||
During prior maintenance, we encountered the follow issues:
1) SeaMicro chassis ATS were not 100% reliable. We will be manually failing over to the appropriate power feeds vs. relying on automatic transfer.
2) OOB switches are single-corded. It was inconvenient to operate for extended periods without OOB access, so we will be restoring OOB access after a brief outage.
3) The 'oobcore1' switch did not have redundant power connected. This was corrected.
Reporter | ||
Comment 2•12 years ago
|
||
We're going to bump up the 7/29 mac mini migration start time. Current schedule is as follows:
7/24 - B-side power is cut from 0500 to 1700 PDT
7/25 - B-side power is cut from 0500 to 1700 PDT
7/26 - No work
7/27 - No work
7/28 - No work
7/29 - 1/3 of the mac mini pool is migrated to B-side power from 0500 to 0900 PDT
7/30 - 1/3 of the mac mini pool is migrated to B-side power from 0600 to 1000 PDT
7/31 - 1/3 of the mac mini pool is migrated to B-side power from 0600 to 1000 PDT
8/01 - A-side power is cut from 0500 to 1700 PDT
8/02 - A-side power is cut from 0500 to 1700 PDT, work is complete
Reporter | ||
Comment 3•12 years ago
|
||
This work has been cancelled by the facility at the request of a neighboring tenant. We will not be allowed to proceed with power maintenance on 7/24 or 7/25, at the very least.
We are regrouping tomorrow with all involved parties in order to create an alternate scheduling proposal. I will provide another update at that time.
Reporter | ||
Comment 4•12 years ago
|
||
It took a few days to get consensus from the other tenants, but we now have our next schedule proposal:
7/29 - B-side power is cut at 0500 PDT
7/30 - B-side power is restored at 1700 PDT
7/31 - 1/3 of the mac mini pool is migrated to B-side power from 0600 to 1000 PDT
8/01 - 1/3 of the mac mini pool is migrated to B-side power from 0600 to 1000 PDT
8/02 - 1/3 of the mac mini pool is migrated to B-side power from 0600 to 1000 PDT
8/03 - No work
8/04 - No work
8/05 - A-side power is cut at 0500 PDT
8/06 - A-side power is restored at 1700 PDT
I'll be reaching out to the impacted groups directly in order to confirm Mozilla approval of this schedule.
Reporter | ||
Comment 5•12 years ago
|
||
This schedule was approved by all tenants and is proceeding as planned.
Reporter | ||
Comment 6•12 years ago
|
||
B-side power is fully de-energized and will be restored within 48 hours.
Reporter | ||
Comment 7•12 years ago
|
||
The B-side electrical installation and expansion work has been approved by the city inspector. We are still on track to restore power by the 48-hour mark.
Reporter | ||
Comment 8•12 years ago
|
||
B-side power is restored at the 48-hour mark. We are now on fully-redundant power until A-side power is de-energized at 0500 PDT on 8/5.
All mac mini's have been moved to B-side. ready for A-side power work from releng side.
Reporter | ||
Comment 10•12 years ago
|
||
This power work has been completed, as scheduled.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Updated•11 years ago
|
Product: mozilla.org → Infrastructure & Operations
You need to log in
before you can comment on or make changes to this bug.
Description
•