1207303 - Infoblox rollout in sfo1/mtv2/pdx1

Status: RESOLVED INVALID

(Infrastructure & Operations :: Change Requests, task)

Description

[Dan Parsons [:lerxst]]

Infoblox rollout. Two parts.

Part 1: Guest VLANs Only
Friday, 9/25, 2PM.
DHCP server for guest VLANs in sfo1/mtv2/pdx1 will be changed from admin hosts to Infoblox. Doing just the guest VLAN to work out any kinks in the process. There is no expected end-user impact. Rollback = simply pointing dhcp back at admin hosts. Point = lerxst; netops point = justdave

Part 2: All remaining VLANs
Friday, 10/2, 6PM.
DHCP server for all remaining VLANs in sfo1/mtv2/pdx1 will be changed from admin hosts to Infoblox. End-user impact is expected to be zero to minimal. Rollback = pointing dhcp back at admin hosts. I will need people in each location to test for me. Coordinating with :melissa for that. Point = lerxst; netops point = justdave

Comment 1

[Peter Radcliffe [:pir]]

Documentation for the MOC if there are problems after rollout?

Comment 2

[Corey Shields [:cshields]]

Hard dependency upon runbook docs for MOC, change docs for EUS & AV ops

Please involve [e-mail] about testing and vetting vidyo rooms after the change.

Comment 3

[Linda Ypulong [:unixfairy]]

Reviewed 9/23 CAB - approved contingent on above

Comment 4

[Dave Miller [:justdave]]

Attachment: ansible config change

Part 1 network-side deployment is complete (sfo1, then pdx1, then mtv2, from 3:45pm though 5:20pm)
guest vlan only
pdx1, sfo1, and mtv2 only

Comment 5

[Dave Miller [:justdave]]

Attachment: Additional ansible change

Made an additional ansible change to fix a logic error in the template that would have caused the default dhcp-helper in offices that did not have infoblox to listen on all interfaces instead of just the ones we had defined.  The broken version did not get deployed anywhere except Toronto.  The error would have resulted in offering dhcp services on vlans where it wasn't needed, and the dhcp server would have ignored those requests anyway (if any occurred) because they came from a vlan it wasn't configured for.

Comment 6

[Dan Parsons [:lerxst]]

In order to finalize documentation, this is being pushed forward to Friday, October 9th.

Comment 7

[Dan Parsons [:lerxst]]

Update: this is happening Thursday, October 8th at 6PM Pacific.

Comment 9

[Dan Parsons [:lerxst]]

Due to the lack of an automated tool for importing static dhcp entries from our ISC-DHCP configuration into Infoblox, the deployment schedule for this project is changing significantly.

For tomorrow (Thursday, October 8th at 6PM), we will be migrating only the following VLANs:

pdx1 ops vlan206
pdx1 cam vlan222
pdx1 corp vlan224

I'd like to do another chunk of VLANs the following Thursday. Depending on how much help I get from Infoblox support, I might have a tool that will allow me to bulk convert all remaining VLANs, and so it's possible the next week will be a huge amount of VLANs, but it might not, again, based on how much support I get.

:justdave, does this work for you? can we do a regular standing thursday 6PM thing for a while?

Comment 10

[Dan Parsons [:lerxst]]

Early draft of Infoblox documentation is here: https://mana.mozilla.org/wiki/display/SYSADMIN/Infoblox+DHCP

Comment 11

[James Barnell [:jbarnell]]

I'm going to chime in here.  

There's a lot of concern in NetOps with not doing this en masse. 

It has the potential to make any type of troubleshooting difficult and confusing.  We're the team who are tasked with responding first and most problems with IP addressing or access are attributed to the network.

My preference:  We need to get the offices done on a schedule that makes a clean cutover for each office.  i.e. PDX on this day, SFO on this day, etc.

Comment 12

[Dan Parsons [:lerxst]]

I intend on keeping the above-linked documentation up to date with which VLANs are where, however if you'd like me to not change any more VLANs over until they're all ready, I am happy to do that, but please convey that desire to cshields as an explanation as to why I'm not at least making incremental progress.

Comment 13

[Corey Shields [:cshields]]

I'm not comfortable doing a mixed environment for prod vlans  (ie: other vlans aside from "guest" which was done first to vet things).  I agree with James on a clean cutover.

Also want to make sure that netops is 'comfortable' with any cutovers, so today's needs to push off entirely.

(In reply to James Barnell [:jbarnell] from comment #11) 
> It has the potential to make any type of troubleshooting difficult and
> confusing.  We're the team who are tasked with responding first and most
> problems with IP addressing or access are attributed to the network.

Dan, please ensure that netops are included in the administrator group of infoblox.

Comment 14

[Corey Shields [:cshields]]

(In reply to Dan Parsons [:lerxst] from comment #10)
> Early draft of Infoblox documentation is here:
> https://mana.mozilla.org/wiki/display/SYSADMIN/Infoblox+DHCP

I'm sorry, this is the only thing I can find in mana for infoblox..  Are there runbooks tied to MOC alerts done elsewhere?

Also need docs on MOC/EUS/AV to do DHCP changes.

Docs are another blocker to this rollout.  Please don't make any changes today, thanks.

Comment 15

[Ashish Vijayaram [:ashish]]

Closing this out. Changes were handled in other bugs.