Infoblox rollout in sfo1/mtv2/pdx1

RESOLVED INVALID

Status

RESOLVED INVALID
3 years ago
2 years ago

People

(Reporter: dparsons, Unassigned)

Tracking

Details

Attachments

(2 attachments)

(Reporter)

Description

3 years ago
Infoblox rollout. Two parts.

Part 1: Guest VLANs Only
Friday, 9/25, 2PM.
DHCP server for guest VLANs in sfo1/mtv2/pdx1 will be changed from admin hosts to Infoblox. Doing just the guest VLAN to work out any kinks in the process. There is no expected end-user impact. Rollback = simply pointing dhcp back at admin hosts. Point = lerxst; netops point = justdave

Part 2: All remaining VLANs
Friday, 10/2, 6PM.
DHCP server for all remaining VLANs in sfo1/mtv2/pdx1 will be changed from admin hosts to Infoblox. End-user impact is expected to be zero to minimal. Rollback = pointing dhcp back at admin hosts. I will need people in each location to test for me. Coordinating with :melissa for that. Point = lerxst; netops point = justdave
Flags: cab-review?
Documentation for the MOC if there are problems after rollout?
Hard dependency upon runbook docs for MOC, change docs for EUS & AV ops

Please involve avops@mozilla.com about testing and vetting vidyo rooms after the change.
Reviewed 9/23 CAB - approved contingent on above
Flags: cab-review? → cab-review+
Created attachment 8666287 [details] [diff] [review]
ansible config change

Part 1 network-side deployment is complete (sfo1, then pdx1, then mtv2, from 3:45pm though 5:20pm)
guest vlan only
pdx1, sfo1, and mtv2 only
Created attachment 8666420 [details]
Additional ansible change

Made an additional ansible change to fix a logic error in the template that would have caused the default dhcp-helper in offices that did not have infoblox to listen on all interfaces instead of just the ones we had defined.  The broken version did not get deployed anywhere except Toronto.  The error would have resulted in offering dhcp services on vlans where it wasn't needed, and the dhcp server would have ignored those requests anyway (if any occurred) because they came from a vlan it wasn't configured for.
(Reporter)

Comment 6

3 years ago
In order to finalize documentation, this is being pushed forward to Friday, October 9th.
(Reporter)

Comment 7

3 years ago
Update: this is happening Thursday, October 8th at 6PM Pacific.
Duplicate of this bug: 1207264
(Reporter)

Comment 9

3 years ago
Due to the lack of an automated tool for importing static dhcp entries from our ISC-DHCP configuration into Infoblox, the deployment schedule for this project is changing significantly.

For tomorrow (Thursday, October 8th at 6PM), we will be migrating only the following VLANs:

pdx1 ops vlan206
pdx1 cam vlan222
pdx1 corp vlan224

I'd like to do another chunk of VLANs the following Thursday. Depending on how much help I get from Infoblox support, I might have a tool that will allow me to bulk convert all remaining VLANs, and so it's possible the next week will be a huge amount of VLANs, but it might not, again, based on how much support I get.

:justdave, does this work for you? can we do a regular standing thursday 6PM thing for a while?
Flags: needinfo?(justdave)
(Reporter)

Comment 10

3 years ago
Early draft of Infoblox documentation is here: https://mana.mozilla.org/wiki/display/SYSADMIN/Infoblox+DHCP
I'm going to chime in here.  

There's a lot of concern in NetOps with not doing this en masse. 

It has the potential to make any type of troubleshooting difficult and confusing.  We're the team who are tasked with responding first and most problems with IP addressing or access are attributed to the network.

My preference:  We need to get the offices done on a schedule that makes a clean cutover for each office.  i.e. PDX on this day, SFO on this day, etc.
(Reporter)

Comment 12

3 years ago
I intend on keeping the above-linked documentation up to date with which VLANs are where, however if you'd like me to not change any more VLANs over until they're all ready, I am happy to do that, but please convey that desire to cshields as an explanation as to why I'm not at least making incremental progress.
I'm not comfortable doing a mixed environment for prod vlans  (ie: other vlans aside from "guest" which was done first to vet things).  I agree with James on a clean cutover.

Also want to make sure that netops is 'comfortable' with any cutovers, so today's needs to push off entirely.

(In reply to James Barnell [:jbarnell] from comment #11) 
> It has the potential to make any type of troubleshooting difficult and
> confusing.  We're the team who are tasked with responding first and most
> problems with IP addressing or access are attributed to the network.

Dan, please ensure that netops are included in the administrator group of infoblox.
(In reply to Dan Parsons [:lerxst] from comment #10)
> Early draft of Infoblox documentation is here:
> https://mana.mozilla.org/wiki/display/SYSADMIN/Infoblox+DHCP

I'm sorry, this is the only thing I can find in mana for infoblox..  Are there runbooks tied to MOC alerts done elsewhere?

Also need docs on MOC/EUS/AV to do DHCP changes.

Docs are another blocker to this rollout.  Please don't make any changes today, thanks.

Updated

3 years ago
Change Request: --- → approved
Flags: cab-review+
Flags: needinfo?(justdave)
Closing this out. Changes were handled in other bugs.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → INVALID
You need to log in before you can comment on or make changes to this bug.