Closed Bug 923592 Opened 10 years ago Closed 10 years ago

MTV Powershut Down (WPR change)

Categories

(Infrastructure & Operations :: Change Requests, task, P2)

Product:
Infrastructure & Operations
Component:
Change Requests
Platform:
x86
macOS
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: avillarde, Unassigned)

References

Details

WPR advised IT during our weekly sync meeting on 01-Oct-13 that the property management company for MTV 650 Castro needs to shut down the entire building for 2 hours to install power stations for electric vehicles.  They have asked Mozilla for best times.  I advised Rob Middleton that we have tree impacting equipment and best to schedule this during a tree closure window.  As much as I knew, 12-Oct and 16-Nov were the next scheduled, but that WPR should raise this to the CAB for firm confirmations.  I'm opening this on behalf of WPR so it's tracked.
Severity: major → normal
Flags: cab-review?
(thanks avillarde for the headsup)

As this will burn all tegra+panda jobs for the duration, I confirm this does need a tree closure. Do you know when this will be done, so we can give developers advance-notice? 

(My hope would be that this power work is done on evening/weekend, to reduce impact to *all* companies in the building, but there's no date/time info specified here. If we *need* to do this poweroutage/treeclosure at a time outside of 12-Oct and 16-Nov, lets get the exact date/time together, and I'll help make a case for it.)
To clarify, it was a WPR planning topic and headsup.  There was no specified date but IT advised to work this into the normal tree closure cycle because of the tegras onsite. WPR is working with building mgmt to coordinate a future time convenient for all tenants.  It's our hope that we arrange that coincide with our cycle.
WPR has confirmed that this is NOT being done this weekend and the vendor is looking at 16-Nov to do this work.  Will update when the 16-Nov date is confirmed.
Due to a major SCL3 network change (see bug 924570) all networking resources will be dedicated to SCL3 during the 16-Nov tree closure. It's expected bug 924570 will take 6-8 hours of Netops time. In the event of any MTV hiccups with service restarts or any networking issues, we may not have coverage to support both simultaneously. It will be recommended to move MTV power shut down to the follow closure window after 16-Nov. If we can avoid overlap with an extended tree closure to accommodate both outages that would be the next preference if we can't move this change past 16-Nov.
Hi Albert,
Thanks for the update.
Property management is wanting to get this done before the end of the year (I think due to City of Mountain View permitting restrictions).
I am sure they are willing to consider another day.  
I think if we recommend a date, we can drive the schedule for the entire building.  Could you discuss in CAB meeting & propose some dates that work best for your team/resources between Nov 18 - Dec 20?

Targeting 2-4 hours, late evening to minimize impact to retail customers, date TBD.
adding :dmoore & some other scheduling constraints and preferences.

(In reply to L Gray from comment #5)
> Targeting 2-4 hours, late evening to minimize impact to retail customers,

After the power is restored, RelEng will need approximately 2 hours to get the trees reopened. So we're looking at a 4-6 hour tree closure in total.

During the post-power-restored time we will likely require hands on support from DCOPS in MTV1. 

Our key experts for this equipment are on the East Coast, so late evening is not as desirable as a weekend.
I will relay information to landlord... weekend will impact retail 1st floor customers but I will ask for earliest time of evening possible.
The landlord has picked Friday 11/22, in the evening for this work.  We are waiting for the exact time.
(In reply to Melissa O'Connor [:melissa] from comment #8)
> The landlord has picked Friday 11/22, in the evening for this work.  We are
> waiting for the exact time.

Unfortunately, this wasn't discuss in the CAB on the 23 Oct :| Will do so next week.
Land lord update;  target shut off time to begin between 20:00 and 22:00 on Friday Nov 22. Estimated time of shut off is 3 hours.
(In reply to L Gray from comment #10)
> Land lord update;  target shut off time to begin between 20:00 and 22:00 on
> Friday Nov 22. Estimated time of shut off is 3 hours.

That's a very wide window for the start and end times -- it means we have to announce tree closure for at least 2000 Fri until 0230 Sat - 6-1/2 hours. (It will take us about 90 minutes to restore everything, and perhaps longer due to scheduling issues.)

Two questions to make planning on our side more possible:
1. Is this a preliminary window, and the landlord will firm up the start time closer to the event? If so, do we have a date for the tighter schedule?

2. is the 3 hours a "no longer than" estimate, or a "if everything goes well" estimate? That will impact how we staff this event from the east coast.
Flags: needinfo?(lgray)
(In reply to Hal Wine [:hwine] (use needinfo) from comment #11)
> (In reply to L Gray from comment #10)
> > Land lord update;  target shut off time to begin between 20:00 and 22:00 on
> > Friday Nov 22. Estimated time of shut off is 3 hours.
> 
> That's a very wide window for the start and end times -- it means we have to
> announce tree closure for at least 2000 Fri until 0230 Sat - 6-1/2 hours.
> (It will take us about 90 minutes to restore everything, and perhaps longer
> due to scheduling issues.)
> 
> Two questions to make planning on our side more possible:
> 1. Is this a preliminary window, and the landlord will firm up the start
> time closer to the event? If so, do we have a date for the tighter schedule?
> 
> 2. is the 3 hours a "no longer than" estimate, or a "if everything goes
> well" estimate? That will impact how we staff this event from the east coast.

Hi Hal, I will check again with the Landlord on both your questions.
Flags: needinfo?(lgray)
At the IT/WPR sync up meeting today it was mentioned that this work MAY slip from Friday evening 11/22 to Saturday evening 11/23. Lisa is waiting to hear back from the landlord.
FINAL Decision on Date & Time for MTV shut down from Land Lord please let me know if you have any questions.
Hello all, 

I want confirm our plan to shut down all power to the building on Friday, November 22, 2013 at 8:00 p.m.  Barring any unforeseen circumstances, we anticipate restoring power by 11:00 p.m.

For your safety and that of your staff members - BE PREPARED - at 8:00 p.m. you will be without power
·        We request that all tenants leave the building by 7:30 p.m.  – It will be the responsibility of each tenant to clear their suite
·        Elevators will be recalled to the first floor starting at 7:30 pm and will  not operate for the duration of the shut down. 
·        Stairwells, suites, garage, restrooms, showers, and corridors do have emergency lighting; however it will not last the duration of the project.  These areas will be completely dark.
·        The garage gate will be locked in the open position.

When power is restored:
·        If there is a specific person that needs to be contacted after the power is restored in order to return to the building to reset equipment or perform a specific task that night, please provide contact information. 

Should we encounter any problem that prevents us from restoring power by 11:00 p.m. please let me know who you would like us to contact (include name, phone and/or email address).
Depends on: 940403
EndUser Services team has already begun the work of turning off hosts.

Seems other teams have turned off servers as well.
Power is off.
(In reply to Adrian Fernandez [:Aj] from comment #16)
> Power is off.

Web QA has powered off (at least) the following:

1) https://wiki.mozilla.org/QA/Execution/Web_Testing/Automation/Selenium_Grid (all the nodes therein)
2) all b2g-x  (includes the target phones) [where x==0 10 to 12]) nodes (both host -- i.e. Mac Mini or Dell workstation in A-team cases)

Raymond and I will be restarting and hopefully ensuring successful restart of the above, starting at 8am PDT.
Power came back on @ around 9:30 pm PST , however, attempting to verify if permanent.
Awaiting landlords confirmation will advise
Just received confirmation. Power has been restored permanently.  Landlord  Test complete.
Thanks
Most of the core infrastructure is back up (IT side, Releng different). For the remaining ones, manual intervention is needed and we are currently working on it.
All set from core infrastructure side, EndUser Services team will verify other office related services tomorrow morning (conference rooms etc).
(In reply to Adrian Fernandez [:Aj] from comment #21)
> Most of the core infrastructure is back up (IT side, Releng different). For
> the remaining ones, manual intervention is needed and we are currently
> working on it.

Trees reopened at 2013-11-23T03:17:49
Status: NEW → RESOLVED
Closed: 10 years ago
Flags: cab-review? → cab-review+
Resolution: --- → FIXED
Product: mozilla.org → Infrastructure & Operations
Change Request: --- → approved
Flags: cab-review+
You need to log in before you can comment on or make changes to this bug.