Closed
Bug 1249112
Opened 9 years ago
Closed 9 years ago
SFO - 243 Mary Ellen Pleasant - Room Controller: Loses wifi connection multiple times a week
Categories
(Infrastructure & Operations Graveyard :: NetOps, task)
Infrastructure & Operations Graveyard
NetOps
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: freshness, Assigned: van)
Details
Crossover from Service Now Ticket:INC0018895
Update from Van:
working with :freshness on the iPad. it looks like the connection is fine right now in the room. pinging it is flawless and we have lots of other users as well as other iPads connected to this AP. RSSI and SNR is way above accepted levels:
Last packet RSSI: -53 dBm
Last packet SNR: 42
wifi1.ops.sfo1.mozilla.net# rfping session-id 32473 verbose
RF-Link Test to 90:8d:6c:18:14:e6 :
Session-Id: 32473
Packets Sent Packets Rcvd RSSI SNR RTT (micro-secs)
------------ ------------ ------- ----- ----------------
20 20 -49 45 975
Index RSSI SNR RTT (micro-secs) Retries Rate (Mb/s)
----- ------------ ------- ---------------- ------- --------
1 -49 46 14041 0 130.0
2 -49 46 216 0 130.0
3 -49 46 560 0 130.0
4 -50 45 179 0 130.0
5 -48 47 269 0 130.0
6 -49 46 252 0 130.0
7 -49 46 468 0 130.0
8 -48 47 233 0 130.0
9 -49 46 297 0 130.0
10 -50 45 250 0 130.0
11 -50 45 179 0 130.0
12 -49 46 305 0 130.0
13 -50 45 171 0 130.0
14 -49 46 196 0 130.0
15 -49 46 641 0 130.0
16 -48 47 188 0 130.0
17 -49 46 424 0 130.0
18 -49 46 223 0 130.0
19 -50 45 170 0 130.0
20 -48 47 244 0 130.0
wifi1.ops.sfo1.mozilla.net#
however checking splunk, we see the iPad is roaming on its own and also causing excessive retransmits.
this is probably one instance when the iPad lost connection:
2/12/16
11:31:45.000 PM
<157>Feb 12 23:31:45 10.251.0.30 AUTHORIZATION-SUCCESS: user: remote@mozilla.com; mac: 90:8d:6c:18:14:e6; author reason: roamed session; ssid: Mozilla; AP 5123/2
2/12/16
11:31:35.000 PM
<157>Feb 12 23:31:35 10.251.0.30 AUTHORIZATION-SUCCESS: user: remote@mozilla.com; mac: 90:8d:6c:18:14:e6; author reason: roamed session; ssid: Mozilla; AP 5123/1
checking trends we see quite a few excessive retransmits:
12.46%
<156>timestamp10.251.0.30 DOT1X_AAA_ERR: Excessive retransmits; client: remote@mozilla.com; mac: 90:8d:6c:18:14:e6; ssid: Mozilla; AP 5123/1
i've been asked to kick the AP after hours, there's a lot of users (31) on it. if it happens again, i suggest looking into replacing the iPad to see if problem persist.
wifi1.ops.sfo1.mozilla.net# show sessions network ap 5123
31 of 392 sessions matched
|
Reporter | |
Comment 1•9 years ago
|
||
Update from Van:
AP rebooted and back online.
AP Flag IP Address Model MAC Address Radio 1 Radio 2 Uptime
---- ---- --------------- ------------ ----------------- ------- ------- ------
5123 o--e 10.251.1.23 WLA532-US ac:4b:c8:0e:fd:40 E 1/1 E149/7 01m37s
|
Assignee | |
Updated•9 years ago
|
Assignee: network-operations → vle
|
|
Assignee | |
Comment 2•9 years ago
|
||
bumped up radio 2 by 1 as the iPad is still flopping between radio 1 and radio 2. this happens if the iPad can't decide which signal is better as it tests for RSSI instead of SNR. checked CCI and it still looks decent in the 5ghz spectrum.
AP: 5123 - wap203.sfo1
Detected BSSID Vendor Class Ch RSSI Age SSID
----------------- -------- ----- ---- ---- --- -----------------
00:0b:86:7b:d5:98 Aruba suspt 149+ -88 76 GenslerCorp
94:b4:0f:8e:d5:90 Unknown suspt 149+ -86 811 GenslerCorp
94:b4:0f:8e:d5:91 Unknown suspt 149+ -86 772 GenslerBYOD
94:b4:0f:8e:d5:92 Unknown suspt 149+ -86 831 GenslerGuest
94:b4:0f:8e:d5:93 Unknown suspt 149+ -86 772 TESTGUEST
9c:1c:12:ed:e7:d0 Unknown suspt 149+ -87 891
ac:4b:c8:0d:83:01 Juniper membr 149 -87 95 Mozilla
ac:4b:c8:0d:83:03 Juniper membr 149 -87 36
ac:4b:c8:0d:83:05 Juniper membr 149 -87 95 openwireless.org
ac:4b:c8:0d:83:07 Juniper membr 149 -87 135 Mozilla Guest
wifi1.ops.sfo1.mozilla.net# show ap status 5123
Flags: o = operational[1], c = configure[0], d = download[0], b = boot[0]
a = auto AP, m = mesh AP, p/P = mesh portal (ena/actv), r = redundant[0]
z = remote AP in outage, i/I = insecure (control/control+data)
u = unencrypted, e/E = encrypted (control/control+data)
Radio: E = enabled - 20MHz channel, S = sentry, s = spectral-data
W/w = enabled - 40MHz wide channel (HTplus/HTminus)
D = admin disabled, U = mesh uplink
IP Address: * = AP behind NAT
AP Flag IP Address Model MAC Address Radio 1 Radio 2 Uptime
---- ---- --------------- ------------ ----------------- ------- ------- ------
5123 o--e 10.251.1.23 WLA532-US ac:4b:c8:0e:fd:40 E 1/1 E149/7 20h58m
wifi1.ops.sfo1.mozilla.net# set ap 5123 radio 2 tx-power 8
splunk logs:
2/17/16
11:28:37.000 PM
<157>Feb 17 23:28:37 10.251.0.30 AUTHORIZATION-SUCCESS: user: remote@mozilla.com; mac: 90:8d:6c:18:14:e6; author reason: new session; ssid: Mozilla; AP 5123/2
host = 10.251.0.30
source = udp:1514
sourcetype = syslog
2/17/16
11:28:37.000 PM
<157>Feb 17 23:28:37 10.251.0.30 AUTHENTICATION-SUCCESS: mac: 90:8d:6c:18:14:e6; ssid: Mozilla; AP 5123/2;
host = 10.251.0.30
source = udp:1514
sourcetype = syslog
2/17/16
11:28:37.000 PM
<157>Feb 17 23:28:37 10.251.0.30 ASSOCIATION-SUCCESS: mac: 90:8d:6c:18:14:e6; ssid: Mozilla; AP5123/Radio2;
host = 10.251.0.30
source = udp:1514
sourcetype = syslog
2/17/16
11:22:26.000 PM
<156>Feb 17 23:22:26 10.251.0.30 DOT1X_AAA_ERR: Excessive retransmits; client: remote@mozilla.com; mac: 90:8d:6c:18:14:e6; ssid: Mozilla; AP 5123/1
host = 10.251.0.30
source = udp:1514
sourcetype = syslog
2/17/16
11:22:11.000 PM
<157>Feb 17 23:22:11 10.251.0.30 ASSOCIATION-SUCCESS: mac: 90:8d:6c:18:14:e6; ssid: Mozilla; AP5123/Radio1;
host = 10.251.0.30
source = udp:1514
sourcetype = syslog
2/17/16
11:05:30.000 PM
<157>Feb 17 23:05:30 10.251.0.30 AUTHORIZATION-SUCCESS: user: remote@mozilla.com; mac: 90:8d:6c:18:14:e6; author reason: new session; ssid: Mozilla; AP 5123/2
host = 10.251.0.30
source = udp:1514
sourcetype = syslog
2/17/16
11:05:30.000 PM
<157>Feb 17 23:05:30 10.251.0.30 AUTHENTICATION-SUCCESS: mac: 90:8d:6c:18:14:e6; ssid: Mozilla; AP 5123/2;
host = 10.251.0.30
source = udp:1514
sourcetype = syslog
2/17/16
11:05:29.000 PM
<157>Feb 17 23:05:29 10.251.0.30 ASSOCIATION-SUCCESS: mac: 90:8d:6c:18:14:e6; ssid: Mozilla; AP5123/Radio2;
host = 10.251.0.30
source = udp:1514
sourcetype = syslog
2/17/16
10:34:52.000 PM
<156>Feb 17 22:34:52 10.251.0.30 DOT1X_AAA_ERR: Excessive retransmits; client: remote@mozilla.com; mac: 90:8d:6c:18:14:e6; ssid: Mozilla; AP 5123/1
host = 10.251.0.30
source = udp:1514
sourcetype = syslog
2/17/16
10:34:37.000 PM
<157>Feb 17 22:34:37 10.251.0.30 ASSOCIATION-SUCCESS: mac: 90:8d:6c:18:14:e6; ssid: Mozilla; AP5123/Radio1;
host = 10.251.0.30
source = udp:1514
sourcetype = syslog
2/17/16
10:12:45.000 PM
<157>Feb 17 22:12:45 10.251.0.30 AUTHORIZATION-SUCCESS: user: remote@mozilla.com; mac: 90:8d:6c:18:14:e6; author reason: new session; ssid: Mozilla; AP 5123/2
host = 10.251.0.30
source = udp:1514
sourcetype = syslog
2/17/16
10:12:45.000 PM
<157>Feb 17 22:12:45 10.251.0.30 AUTHENTICATION-SUCCESS: mac: 90:8d:6c:18:14:e6; ssid: Mozilla; AP 5123/2;
host = 10.251.0.30
source = udp:1514
sourcetype = syslog
2/17/16
10:12:45.000 PM
<157>Feb 17 22:12:45 10.251.0.30 ASSOCIATION-SUCCESS: mac: 90:8d:6c:18:14:e6; ssid: Mozilla; AP5123/Radio2;
host = 10.251.0.30
source = udp:1514
sourcetype = syslog
2/17/16
10:00:59.000 PM
<156>Feb 17 22:00:59 10.251.0.30 DOT1X_AAA_ERR: Excessive retransmits; client: remote@mozilla.com; mac: 90:8d:6c:18:14:e6; ssid: Mozilla; AP 5123/1
|
|
Reporter | |
Comment 3•9 years ago
|
||
Since the radio 2 increase, Connection timeouts and disconnects have not been reported or seen by technicians. Thank you very much Van, closing this bug.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
|
|
Reporter | |
Comment 4•9 years ago
|
||
INC0019264 has been created.
The same behavior was discovered on 3/15/16 by Sylvie during a Vidyo Call. Is there anything else we can do to improve connectivity in this room? Do we need to add another AP closer to the room? Unfortunately, the room is located in the corner of SFO2, and has four walls of concrete around it.
Status: RESOLVED → REOPENED
Flags: needinfo?(vle)
Resolution: FIXED → ---
|
|
Assignee | |
Comment 5•9 years ago
|
||
:freshness, i think we discussed trying a different iPad if the issue persists? when i ping the iPad, im getting excellent results. the signal strenght/noise to the iPad is also way above threshold.
wifi1.ops.sfo1.mozilla.net# show sessions network mac-addr 90:8d:6c:18:14:e6
1 of 172 sessions matched
User Name SessID Type Address VLAN AP/Rdo
--------------------- ------ ----- -------------------- -------------- -------
remote@mozilla.com 39546* dot1x 10.251.25.64,V6 corp L 5123/2 <~~~ radio 2 means 5ghz
wifi1.ops.sfo1.mozilla.net# rfping session-id 39546 verbose
RF-Link Test to 90:8d:6c:18:14:e6 :
Session-Id: 39546
Packets Sent Packets Rcvd RSSI SNR RTT (micro-secs)
------------ ------------ ------- ----- ----------------
20 20 -50 44 216 <~~~~~ -50 RSSI/44 SNR is extremely good and
Index RSSI SNR RTT (micro-secs) Retries Rate (Mb/s)
----- ------------ ------- ---------------- ------- --------
1 -51 44 0 0 130.0
2 -52 43 268 0 130.0
3 -51 44 207 0 130.0
4 -51 44 252 0 130.0
5 -51 44 188 0 130.0
6 -51 44 251 0 130.0
7 -51 44 179 0 130.0
8 -51 44 268 0 130.0
9 -51 44 208 0 130.0
10 -50 45 260 0 130.0
11 -51 44 305 0 130.0
12 -49 46 179 0 130.0
13 -50 45 197 0 130.0
14 -49 46 279 0 130.0
15 -50 45 277 0 130.0
16 -50 45 170 0 130.0
17 -50 45 188 0 130.0
18 -50 45 226 0 130.0
19 -50 45 214 0 130.0
20 -50 45 198 0 130.0
wifi1.ops.sfo1.mozilla.net# rfping session-id 39546 verbose
RF-Link Test to 90:8d:6c:18:14:e6 :
Session-Id: 39546
Packets Sent Packets Rcvd RSSI SNR RTT (micro-secs)
------------ ------------ ------- ----- ----------------
20 20 -50 44 261
Index RSSI SNR RTT (micro-secs) Retries Rate (Mb/s)
----- ------------ ------- ---------------- ------- --------
1 -50 45 0 0 130.0
2 -51 44 171 0 130.0
3 -51 44 214 0 130.0
4 -51 44 297 0 130.0
5 -50 45 170 0 130.0
6 -50 45 242 0 130.0
7 -50 45 279 0 130.0
8 -51 44 251 0 130.0
9 -51 44 197 0 130.0
10 -50 45 171 0 130.0
11 -50 45 296 0 130.0
12 -50 45 878 0 130.0
13 -50 45 233 0 130.0
14 -50 45 305 0 130.0
15 -50 45 297 0 130.0
16 -50 45 214 0 130.0
17 -49 46 225 0 130.0
18 -49 46 296 0 130.0
19 -50 45 286 0 130.0
20 -51 44 189 0 130.0
it looks like the iPad for whatever reason is sometimes choosing to use the 2.4ghz network over the 5ghz even though the 5ghz signal is much better.
13.03%
<156>timestamp10.251.0.30 DOT1X_AAA_ERR: Excessive retransmits; client: remote@mozilla.com; mac: 90:8d:6c:18:14:e6; ssid: Mozilla; AP 5123/1 <~~~ radio 1 means 2.4ghz
adding another AP is not practical as it requires an additional network drop and may also introduce cross/co-channel interference.
Flags: needinfo?(vle)
|
|
Assignee | |
Updated•9 years ago
|
Flags: needinfo?(mrichards)
|
|
Reporter | |
Comment 6•9 years ago
|
||
Thanks so much for the info! That's extremely helpful. Even though the iPad's receiving great signal strength, it still dropped around 3pm yesterday. In hopes of tracking down the issue, the problematic iPad has been replaced with an iPad Air 2, currently running iOS 9.1.2
MAC address:24:A0:74:6F:EB:0D
We are currently investigating possible connection errors between the iPad and the MC3, which would give us the same connection error message we're seeing.
Flags: needinfo?(mrichards)
|
|
Assignee | |
Comment 7•9 years ago
|
||
5:07 PM <Freshness> It's not an AP issue, it's an iOS issue
closing bug.
Status: REOPENED → RESOLVED
Closed: 9 years ago → 9 years ago
Resolution: --- → FIXED
|
|
Assignee | |
Comment 8•9 years ago
|
||
8:43 AM <jbarnell> can you reopen the bug on ipads? we think but we're not sure its related to the IOS version and i think we need data to verify before it goes resolved
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
|
|
Assignee | |
Comment 9•9 years ago
|
||
havent heard anything in ~3 months. let me know if this still needs prodding.
Status: REOPENED → RESOLVED
Closed: 9 years ago → 9 years ago
Resolution: --- → FIXED
|
||
Updated•3 years ago
|
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•