1158942 - The Unbearable Timeouts

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: NetOps: Office Wireless, task)

Description

[Selena Deckelmann :selenamarie :selena]

This is me, pinging www.mozilla.com from the Portland office just now: 

--- static-redirect.zlb.phx.mozilla.net ping statistics ---
274 packets transmitted, 228 received, 16% packet loss, time 273558ms
rtt min/avg/max/mdev = 34.357/45.939/587.842/59.964 ms

Pinging something outside of the Mozilla network was more like 20% packet loss. 

This is ongoing, every day. I can't stay logged into IRC. Fetching large files from production and stage systems times out and sometimes requires multiple attempts. I login via VPN and get kicked out during periods of packet loss multiple times a day.

Logging into the wireless sometimes takes 10-15 minutes.  I get a cup of coffee after I plugin here at work (around 7:30-8am) in hopes that my network will be working by the time I get back. Vidyo times out in the office. 

Help!

Comment 1

[Selena Deckelmann :selenamarie :selena]

From the time I hit "submit" to when the page rendered was about 60 seconds. During that time I also experienced lag in SSH sessions.

Comment 3

[Selena Deckelmann :selenamarie :selena]

Traceroutes: 

selena@taiping:~$ traceroute www.mozilla.com
traceroute to www.mozilla.com (63.245.217.20), 30 hops max, 60 byte packets
 1  10.248.32.1 (10.248.32.1)  1.794 ms  1.800 ms  1.852 ms
 2  100.xe-1-2-0.mpr1.pdx1.us.above.net (208.185.170.41)  1.950 ms  2.008 ms  2.092 ms
 3  ae2.cr1.sjc2.us.zip.zayo.com (64.125.20.21)  16.991 ms  17.653 ms  18.308 ms
 4  v11.ae29.cr1.lax112.us.zip.zayo.com (64.125.31.1)  27.957 ms  28.000 ms  28.060 ms
 5  ae7.mpr3.phx2.us.zip.zayo.com (64.125.31.110)  36.698 ms  36.987 ms  37.182 ms
 6  64.124.201.178.mozilla.com (64.124.201.178)  37.751 ms  34.464 ms  33.941 ms
 7  xe-1-0-1.core2.phx1.mozilla.net (63.245.216.26)  35.812 ms  35.856 ms  35.948 ms
 8  * * *
 

selena@taiping:~$ traceroute hg.mozilla.org
traceroute to hg.mozilla.org (63.245.215.102), 30 hops max, 60 byte packets
 1  10.248.32.1 (10.248.32.1)  15.208 ms  15.585 ms  15.637 ms
 2  100.xe-1-2-0.mpr1.pdx1.us.above.net (208.185.170.41)  15.714 ms  16.054 ms  16.095 ms
 3  ae3.mpr2.pdx1.us.zip.zayo.com (64.125.30.22)  19.476 ms  19.573 ms  19.692 ms
 4  ae4.cr2.sjc2.us.zip.zayo.com (64.125.22.1)  65.775 ms  65.771 ms  65.883 ms
 5  ae7.mpr2.pao1.us.zip.zayo.com (64.125.25.138)  35.721 ms  35.697 ms  35.829 ms
 6  64.125.170.34.t00539-05.above.net (64.125.170.34)  35.958 ms  16.832 ms  16.181 ms
 7  xe-0-0-1.border1.scl3.mozilla.net (63.245.219.130)  16.365 ms  16.908 ms  19.325 ms
 8  v-1026.core1.scl3.mozilla.net (63.245.214.70)  19.487 ms  19.655 ms  19.875 ms
 9  * * *


Then I had a bunch of DNS timeouts.

And I tried this: 

selena@taiping:~$ traceroute www.google.com
www.google.com: Name or service not known
Cannot handle "host" cmdline arg `www.google.com' on position 1 (argc 1)
selena@taiping:~$ traceroute www.google.com
traceroute to www.google.com (216.58.216.164), 30 hops max, 60 byte packets
 1  10.248.32.1 (10.248.32.1)  0.975 ms  1.002 ms  1.499 ms
 2  100.xe-1-2-0.mpr1.pdx1.us.above.net (208.185.170.41)  2.046 ms  2.718 ms  2.801 ms
 3  ae8.mpr1.sea1.us.zip.zayo.com (64.125.30.25)  6.793 ms  6.829 ms  7.412 ms
 4  zayo-google.sea1.us.zip.zayo.com (64.125.12.70)  7.874 ms  7.942 ms  8.227 ms
 5  209.85.249.34 (209.85.249.34)  8.309 ms  8.395 ms  8.600 ms
 6  216.239.51.161 (216.239.51.161)  8.720 ms  4.909 ms  5.682 ms
 7  sea15s02-in-f4.1e100.net (216.58.216.164)  6.923 ms  7.117 ms  7.209 ms

Comment 4

[Andrew Sliwinski:thisandagain]

I'm not able to view bug 1158922. Can you please grant access so we can track progress?

Comment 5

[Dave Miller [:justdave]]

(In reply to Andrew Sliwinski:thisandagain from comment #4)
> I'm not able to view bug 1158922. Can you please grant access so we can
> track progress?

I moved it from infra to moco.