1472833 - [Tracker] Scheduled Tree Closing Window, Sat July 14th 2018, 06:00 - 15:00 PT

Status: RESOLVED FIXED

(Infrastructure & Operations :: MOC: Service Requests, task)

Description

[Keegan Ferrando [:fauweh]]

Placeholder bug for July 14th Tree Closing Window.

Bug 1468437 / CHG0013025 - OpenVPN maintenance (10m) - Tenative start 08:00
Bug 1471612 - Bugzilla DB changes (8+ hours) - Tentative start 08:30
Bug 1471319 - hgmo migration from SCL3 to MDC1 (~4 Hours) - Tentative start 08:30

HG and BMO work can be performed simultaneously and after VPN work is completed successfully.

Comment 1

[Kendall Libby [:fubar] (he/him)]

I assume times are Pacific?

Comment 2

[Maryann Cristofi [:mcristofi]]

(In reply to Kendall Libby [:fubar] from comment #1)
> I assume times are Pacific?

Yes, PST.

Comment 3

[Keegan Ferrando [:fauweh]]

Hey Bob, do you guys have a time you plan to start your BMO work on 7/14? We want to ensure our VPN work doesn't impact your timing.

Comment 4

[Bob Micheletto [:bobm]]

(In reply to Keegan Ferrando [:fauweh] from comment #3)
> Hey Bob, do you guys have a time you plan to start your BMO work on 7/14? We
> want to ensure our VPN work doesn't impact your timing.

It is my intention to start at 09:00 AM US/Eastern (06:00 AM US/Pacific).

Comment 5

[Ryan C [:ryanc] (UTC-4)]

Looks like this starts during Justin's shift

Comment 6

[Keegan Ferrando [:fauweh]]

Final schedule will be:

06:00 - ~14:00 PST:
Bug 1471612 - Bugzilla DB changes (~8 hours)

09:00 - ~13:00 PST:
Bug 1471319 - hgmo migration from SCL3 to MDC1 (~4 Hours)

~14:00 PST - 15:00 PST:
Bug 1468437 / CHG0013025 - OpenVPN maintenance (~10 minutes)

Comment 7

[Kim Moir [:kmoir] ET]

When I used to do tree closures in the past when I worked in releng, I followed the process here

https://moz-releng-docs.readthedocs.io/en/latest/procedures/TCW_Process.html#execution-of-tcw				

The previous MOC manager Linda used to create a spreadsheet with all the tasks that we needed to accomplish and then we would mark them as green when they were complete so other people could see what status was by looking at the spreadsheet.  I'm not sure what the process is now but I setup a quick spreadsheet to that end for tomorrow, if you find it helpful should all be able to edit it.

https://docs.google.com/spreadsheets/d/114gWayI6rE1Ky3aA-xBtherefruFPdc6aUzzb15015g/edit#gid=0

She also would have "rest periods" between tasks so we could assess after 15-30 minutes after a migration completed if there were errors or system failures that needed to be addressed.  This mitigated the pain of figuring out what changed the problem, instead of, for example, making many network changes in quick succession and then trying to figure out which one was the root cause of the problem.

Comment 8

[Keegan Ferrando [:fauweh]]

(In reply to Kim Moir [:kmoir] ET from comment #7)
> When I used to do tree closures in the past when I worked in releng, I
> followed the process here
> 
> https://moz-releng-docs.readthedocs.io/en/latest/procedures/TCW_Process.
> html#execution-of-tcw				
> 
> The previous MOC manager Linda used to create a spreadsheet with all the
> tasks that we needed to accomplish and then we would mark them as green when
> they were complete so other people could see what status was by looking at
> the spreadsheet.  I'm not sure what the process is now but I setup a quick
> spreadsheet to that end for tomorrow, if you find it helpful should all be
> able to edit it.
> 
> https://docs.google.com/spreadsheets/d/114gWayI6rE1Ky3aA-
> xBtherefruFPdc6aUzzb15015g/edit#gid=0
> 
> She also would have "rest periods" between tasks so we could assess after
> 15-30 minutes after a migration completed if there were errors or system
> failures that needed to be addressed.  This mitigated the pain of figuring
> out what changed the problem, instead of, for example, making many network
> changes in quick succession and then trying to figure out which one was the
> root cause of the problem.

We discussed this in our last two weekly Change Management meetings and elected to skip the spreadsheet as the BMO/HG changes will happen simultaneously and Greg will work with those teams to perform his VPN work once they are confident the BMO/HG teams' work has been completed and settled. As those work events could take less/more time, the VPN work start will be dynamic. Maryann also met directly with Kendall and Jordan directly to address any concerns there and Dylan represented the BMO changes in our CAB meeting. Justin Lazaro will be the MOC tech on duty to handle any communications and to work with Sheriffs to close the trees.

Other than that, emails have been sent to the same release/IT distros that have been used for the past several TCW's so let me know if we're missing people (I'm not completely sure the relationship of the recipients in our email and your referenced groups "dev-planning, dev-tree-management, dev-platform")

We will keep the spreadsheet you provided updated. For future Tree Closing Windows, if there is any process improvements that you recommend, please let us know to ensure we are successful in meeting your business needs in the future.

Comment 9

[Keegan Ferrando [:fauweh]]

TCW has concluded. Due to time overrun on the BMO migration, the VPN work was cancelled and will be rescheduled for another time, TBD. 

BMO database maintenance and HG migration were successfully completed.

Timeline:

19:58 PT - BMO service restored (final validation in process)
20:22 PT - HG migration complete
20:24 PT - Trees reopen
21:15 PT - HG network issues from mdc2 clients
22:01 PT - Final HG networking issues resolved

Comment 10

[Kim Moir [:kmoir] ET]

In reference to comment #8, my apologies for jumping in at the last minute and suggesting a process change. I did not attend the change management meetings and wasn't sure how the TCW process had changed recently.  I just found them extremely helpful in the past when I participated in TCW as part of releng.  But it was not helpful of me to suggest a process change without understanding all existing context. Thank you to the MOC for all the work of the team during the TCW and I look forward to working with you going forward.

dev-planning, dev-platform are mailing lists for firefox developers

Comment 11

[Justin Lazaro [:jlaz] (use needinfo)]

No problemo :kmoir! Closing as this has already passed.