947003 - risks and contingency tracker for move of tegras/foopies/web-servers from mtv1 to scl3

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[hwine]

To avoid cluttering the other actionable trackers, we'll list each major risk here (perhaps as separate bugs). We'll put in the mitigation options and recovery options there.

This will also give us a place to record decisions on which risks we're accepting "as is".

Comment 1

[hwine]

One risk is inability to test new SCL3 vlan setup in advance of physical move.

bug 949739 mitigates by moving the staging tegras/foopies in advance, so relops has ~4 days to test and correct networking setup.

Comment 2

[Amy Rich [:arr] [:arich]]

Staging foopies and tegras have been moved and tested (including testing staging tegra boards in production).  It was extremely helpful to have done this, because we found a number of issues that would have blocked us and caused us to overrun our planned window on the day of the move.

* PDU assignmenets didn't work out initially, were adjusted
* VLAN284 network segment not configured on dhcp servers - added to global.conf
* VLAN security zone didn't have any NAT to reach outbound (hg, ftp, etc) - nat added to VLAN
* CNAMEs did not exist for puppet and repos on the tegra VLAN - created
* old ganglia install on foopies meant that puppet would not run successfully - ganglia removed from puppet config
* needed help from uberj to make inventory/DNS updates go more smoothly
* foopies confirmed to require a kickstart since the fqdn changes.  Puppet certs must be regenerated

At this point, we still have been unable to test the webheads since there is no staging environment for them, but getting all of hte other kinks worked out will allow us to focus on those during the actual tree closing window on the 11th.

Comment 3

[Amy Rich [:arr] [:arich]]

Move complete.