1591569 - The Mozilla network cuts out every ~30 minutes.

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: NetOps: Office Wireless, task)

Description

[Sabrina Rudkin]

The Mozilla network cuts out every ~45 minutes. I can reproduce it by:

• Sign into Mozilla IRC using Adium on macOS (10.14.6 Mojave)
• Wait ~45 minutes
• Adium will disconnect and reconnect to IRC. This will happen on that regular interval (and creates a disruptive notification each time)

I've tried this under the following conditions:

• The wifi at my desk on SFO7
• The wifi in the SFO1 Commons
• The ethernet port (with wifi disabled) at my desk on SFO7
• Another coworker's macOS computer (near my desk on wifi on SFO7)
• Another IRC client, Textual, on my machine over wifi.

Textual adds the additional error message, "No route to host".

Given these scenarios, I believe this is happening at the network level (i.e. not wifi) or the OS level.

I have also reproduced similar behavior when being connected to my Android devices over Wifi using ADB. In other apps, I have not had significant trouble though I occasionally will see error pages (i.e. not increased loading times) when trying to load pages in Firefox (I assume it happens only if I try to load a page during the half second or so the wifi cuts out).

This is a reopen of INC0028157 given the provided work-around did not work there.

Comment 1

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

I'm the original reporter: please let me know if you need assistance in debugging!

Comment 2

[Van Le [:van]]

i am not sure i understand the question. are you experiencing complete wifi outages on your laptop? or are you saying IRC disconnects randomly for you?

on the network level, i don't see any logs that say you're actually disconnecting from the network.

Comment 3

[Van Le [:van]]

on the network level, i don't see any logs that say you're actually disconnecting from the network.

what i meant is i don't see logs indicating that you've been dropped or any strange events for [e-mail]. the last events were from 2 weeks ago.

Match Events
Event Time Descending From AP To AP Reason
10/14/2019 5:27 PM PDT aruba-wap709.ops.sfo1 aruba-wap709.ops.sfo1 Band Steer
10/14/2019 5:27 PM PDT aruba-wap710.ops.sfo1 aruba-wap711.ops.sfo1 Sticky Client
10/14/2019 1:13 PM PDT aruba-wap709.ops.sfo1 aruba-wap710.ops.sfo1 Sticky Client

Comment 4

[Van Le [:van]]

:srudkin, i can't see the ticket INC0028157 in service-now. can you cc me on it so i can get the full history?

Your search - INC0028157 - did not match any documents

Suggestions:

Make sure all words are spelled correctly
Try different, more general, or fewer keywords

Comment 5

[Van Le [:van]]

it looks like your laptop is losing wifi completely, if i'm not mistaken.

:srudkin, since you're EUS on site, has there been any other reports of wifi issues in that area? (i haven't received or heard of any reports.) if it's just michael's laptop, it could be a hardware issue on his laptop, similar to :ashish's issue in MTV2. do you have another spare laptop to let him test?

Comment 6

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

(In reply to Van Le [:van] from comment #2)

i am not sure i understand the question. are you experiencing complete wifi outages on your laptop? or are you saying IRC disconnects randomly for you?

I believe I'm experiencing very brief (1-2 seconds?) entire network outages on my laptop around every 45 minutes. In practice, this looks like:

• IRC disconnects and reconnects
• I lose connection to Android devices over wifi via ADB and have to manually reconnect
• If I happen to be trying to load a web page in Firefox, I get an error page. Trying to refresh will usually result in an error page for the next few seconds until my connect in restored

I have not experienced issues in other apps though they may have more robust error handling capabilities that explain it.

(In reply to Van Le [:van] from comment #5)

it looks like your laptop is losing wifi completely, if i'm not mistaken.

I have tried this on both wifi and on an ethernet cable so I don't think it's my wifi. It's possible it's a larger hardware component (network adapter?) but when I tested on another coworker's computer, they experienced the same outages on IRC as I did so I'm not so sure it's my hardware (though we're both using Macbook Pros).

Has IT or NetOps been able to reproduce the issue, using the steps I provided in comment 0? Alternatively, I can try it on a loaner macOS machine (I don't know of Linux/Windows software that is guaranteed to have the same behavior). If we can't reproduce, it may be the hardware.

Comment 7

[Van Le [:van]]

:nfette, can you install the Aruba sensor over by :mcomella's desk the next time you're in the office?

Comment 8

[Van Le [:van]]

:mcomella, our sr. network engineer is on site in SFO1. can you ping her (nadia fette/nfette) on slack to sync up so she can take a look at your issue when you're in your area?

Comment 9

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

I reached out to nfette on Slack: she recommended I try to reproduce on a loaner laptop.

I was able to reproduce on the loaner laptop (with Adium), a mid-2015 MBP running Mojave. Notes:

• This reproduces every 30min almost exactly: 2:15, 2:45, 3:15, 3:45, etc. I noticed my personal laptop has the same pattern.
• In the Network system setting, the IP address never changes and the 802.1X "Connect Time" does not reset no 0 (I had a theory that the DHCP server was reassigning the IP when it expired)
• The times that the loaner reproduces does not correlate with the times that my laptop reproduces

Around 13:20 I disabled "auto-join" of the ateam network on my personal laptop such that "Mozilla" is the only network my computer will auto-join and I've been unable to reproduce the issue on this laptop since then (it's now 15:45, 2h 25min later). This is surprising behavior because I was previously able to reproduce on the wired network. fwiw, the loaner laptop is configured to auto-join "Mozilla Guest".

I'll keep the NI so that:

• I'll let you know if I am able to reproduce on my personal laptop again
• I disabled "auto-join" for other networks on the loaner laptop: I'll let you know if I'm able to reproduce there

Comment 10

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

The loaner laptop, even after disabling "auto-join" on networks besides "Mozilla", is continuing to reproduce so perhaps it has been a fluke that I'm no longer having issues on my personal laptop. I'm clearing NI: please let me know if you need more information.

Comment 11

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

This issue started to reproduce again this morning on my personal laptop (btw, by "personal laptop", I mean the laptop assigned to me for work)

Comment 12

[Van Le [:van]]

it shows your laptop is using a link local ipv6 address. will you be in the office tomorrow?

Comment 13

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Yes, I'm in the office today.

Comment 14

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

I ran a test locally:

• while true; do netstat -rn | head -n 20 && sleep 1; done: I looked for anything suspicious in the routing tables, in particular if the gateway expired (it did not; the expiration also doesn’t sync with the disconnects)
• while true; do traceroute 8.8.8.8; echo ""; sleep 1; done: I looked for anything suspicious (I didn’t find anything)

It’s possible the disconnection happened quickly enough that neither command caught the error: ideally, we could test a persistent connection.

Comment 15

[Nadia]

I worked with Micheal today and yesterday over Slack starting with a few questions do try and identify the root cause of these disconnects as he stated that he could reproduce the issue on loaner laptops as well as a co-workers laptop having the same issue
Asked him a set of questions

1. Did all of the devices have the same OS and applications installed? This could more than likely be an app that is interfering with your network adapter, however we do have some wireless issues in your area, however not quite as bad as disconnecting your network every 30 minutes. Did all 3 laptops disconnect with the same frequency? The Aruba Sensor should capture any issues on the WIFI once it is functioning.
2. Did you test the ethernet port on all three devices and each case when testing the ethernet port did you turn your WIFI off? If you did not turn the WIFI off please test the ethernet port again with the WIFI turned off. Did you test the 3 loner laptops on the ethernet port with the wifi turned off?
3. When did this issue begin occurring with your original laptop? What changes or upgrades occurred at the same time, any new app’s installed when the issue began?
   I also had been doing a continue ping on his laptop yesterday for over 2 hours with minimal packet loss
   . --- 10.251.25.213 ping statistics ---
   8472 packets transmitted, 8428 packets received, 0.5% packet loss
   round-trip min/avg/max/stddev = 2.848/46.504/450.830/45.935

I was advised the following:
Did all of the devices have the same OS and applications installed?
They are all on Mojave (not sure if they have the latest security patches) and all used Adium to repro (which should be the same because it hasn’t been updated in a long time).
They all don’t have the same applications installed: my coworker and I very likely diverge and the loaner laptop looks like a mostly clean install of Mozilla’s macOS.
Did all 3 laptops disconnect with the same frequency?
The 2 laptops I was in control of disconnected every ~30min. I didn’t verify the timing for my coworker (who has an adjacent desk though I’ve reproduced this downstairs as well)
Did you test the ethernet port on all three devices and each case when testing the ethernet port did you turn your WIFI off?
I tested on my laptop with ethernet and the wifi turned off and I was able to reproduce. I did not test on anything but wifi for the loaner or my coworker’s laptop.
When did this issue begin occurring with your original laptop? What changes or upgrades occurred at the same time, any new app’s installed when the issue began?
I noticed the issue prominently when IRCCloud was no longer supported so I had to switch to a local IRC client: the two clients I tried (Adium, Textual) had this problem immediately when I started using them. I’ve noticed my connection to Android devices over Wifi has been disconnecting regularly for a while (more than a year) and all 3 devs in the office have noticed this. I had assumed it was a problem with the ateam network (because I never tried on the Mozilla network). It’s gotten better recently though this could be because the ADB software for connecting to Android devices is more fault tolerant
8472 packets transmitted, 8428 packets received, 0.5% packet loss
The disconnect time appears to be around a second so I’d not be surprised if it happened and it’s just not noticeable above the noise: perhaps we should try over a wired network?
That being said, it could be an internal network -> external network connection issue. One thing that stood out to me in Textual’s error message was, “No route to host.” I’m not sure how network internals work but it’s notable to me that this isn’t a typical network error which usually results in longer latency. Perhaps it would be helpful to loop over traceroute, or another “route to host” application to see if it returns negative results (edited)
Is there any follow-up action items you have for me? I could try the loaner on ethernet but I’m not sure it’ll produce any different results. I also have a linux machine: I could try a few apps to see if it can reproduce the problem

Comment 16

[Nadia]

I sat at Michael's desk for over 1 hour today on WIFI with no network disconnects at all, however I do not run any IRC chat clients. At this point this is pointing to an issue with the application not the network since he stated that the issue immediately started after our IRC cloud was decommissioned and started using local clients "Adium and Textual". Since this is happening on both his WIFI and Ethernet adapter it appears that both these applications are doing something to his adapters and causing his disconnects. I have had a co-worker mention he stopped using Adium for the same issue.
This would also explain why his co-worker is complaining of the same issue as they use the same application.
At this point it is very obvious that there is nothing wrong with the Mozilla Network but the applications themselves so I am going to close the case. I am not sure if EUS can assist with this matter it just may be a case of finding a Chat client that doesn't cause the network adapters to get confused.