1150113 - NA wireless upgrade

Status: RESOLVED FIXED

(Infrastructure & Operations :: Change Requests, task)

Description

[Van Le [:van]]

Exactly similar to 1131971.

date, time, duration of maintenance
April 4, 10PM PST, 12 hours

    system(s) affected
North America (MTV, SFO, TOR, YVR) wireless controllers and APs

    end-user impact
wireless will be unavailable for ~30min (60min) while the APs reboot to install their new code release

    maintenance plan and timeline
https://mana.mozilla.org/wiki/display/NETOPS/Juniper+wireless+upgrade#Juniperwirelessupgrade-WLC%28MSS:MobilitySystemSoftware%29

    rollback plan / rollback point
If any issue is noticed (instability, connections issues, etc...) restart the controllers to their backup partition and investigate

    notification mechanisms
Oncall, whistlepig

    who will be point, who else will be involved
Van, Arzhel

Comment 1

[Van Le [:van]]

we are particularly interested in these 2 bug fixes:

WLAs rebooted because the mobility domain members gradually lost connectivity to each other. [PR 1012899]

When you add new WLAs in a mobility domain, secondary seed becomes active even though the primary seed is
active during mobility domain synchronization. Therefore, there are two active seed controllers in the network at the
same time causing the WLAs to reboot. [PR 1035248]

Comment 2

[James Barnell [:jbarnell]]

Ashlee we are planning to do this on 4/4 beginning at 10:00PM PDT.  This impacts the following offices:

San Francisco
Mountain View
Portland
Vancouver
Toronto

Notification:

We have identified a serious problem with the software running the wireless network in your office.  In order to remediate this issue we will be upgrading all Access Points (AP) and Wireless Controllers on Saturday, April 4 beginning at 10:00PM PDT.  The outage window is planned for a total of 12 hours during this window your office will experience intermittent access to wifi.

Comment 3

[Van Le [:van]]

this was completed successfully on Saturday night and I came on site Sunday to confirm. Please let me know if there are any reported issues.