Closed
Bug 575325
Opened 14 years ago
Closed 14 years ago
External network is slow
Categories
(Infrastructure & Operations Graveyard :: NetOps, task)
Tracking
(Not tracked)
RESOLVED
DUPLICATE
of bug 571922
People
(Reporter: clouserw, Assigned: dmoore)
References
Details
Attachments
(2 files)
Normally I'd blame this on Comcast but after some whining on IRC it sounds like it affects other people too. For the past few weeks, all my network traffic to mozilla servers has been slow. Everything from DNS to actually transferring files. Loading bugzilla pages takes several seconds, static images off people.mozilla.org are slow, it took almost 20 seconds to load the header image on air.mozilla.com this morning. I'm using a mix of nightlies and chrome so I don't think it's my browser. Traceroute doesn't show anything crazy, average 30ms to get to mozilla. I'm not sure what's going on. I'm filing this bug to see if it's a known issue and to see if anyone else has the problem. Did anything change about 2 or 3 weeks ago? Is there anything else I can run to troubleshoot this?
|
||
Comment 1•14 years ago
|
||
(In reply to comment #0) > Did anything change about 2 or 3 weeks ago? Like the whole network going down?
|
||
Comment 2•14 years ago
|
||
Also an issue for me and I'm still seeing high latency/packet loss to cwie-llc.edge8:
> shiny:~ oremj$ mtr --report boris.mozilla.com
> HOST: shiny.local Loss% Snt Last Avg Best Wrst StDev
> 1. 192.168.1.1 60.0% 10 1.3 2.1 0.9 4.6 1.7
> 2. 96.132.240.1 0.0% 10 36.0 30.1 21.4 38.1 5.2
> 3. 68.85.149.113 0.0% 10 36.2 29.1 22.1 36.2 4.0
> 4. te-9-2-ar01.beaverton.or.bve 0.0% 10 36.8 31.5 22.9 53.6 8.8
> 5. te-1-0-0-0-cr01.sacramento.c 0.0% 10 37.1 37.5 35.4 47.5 3.6
> 6. pos-0-7-0-0-cr01.sanjose.ca. 0.0% 10 39.4 39.8 38.5 42.8 1.5
> 7. xe-10-2-0.edge1.sanjose1.lev 0.0% 10 57.1 55.0 37.4 132.7 32.3
> 8. ae-1-69.edge8.sanjose1.level 0.0% 10 37.8 42.9 37.8 63.2 9.5
> 9. cwie-llc.edge8.sanjose1.leve 10.0% 10 205.5 95.0 38.0 279.9 89.4
> 10. v8.core2.sj.mozilla.com 0.0% 10 39.7 40.0 38.4 43.0 1.6
> 11. v2.core3.sj.mozilla.com 0.0% 10 40.3 41.5 38.5 53.0 4.2
> 12. boris.mozilla.com 0.0% 10 41.3 40.0 38.5 41.3 0.9|
|
||
Comment 3•14 years ago
|
||
For comparison:
> shiny:~ oremj$ mtr --report google.com | perl -pne 's/^/> /'
> HOST: shiny.local Loss% Snt Last Avg Best Wrst StDev
> 1. 192.168.1.1 80.0% 10 1.8 1.3 0.8 1.8 0.8
> 2. 96.132.240.1 0.0% 10 48.4 31.2 24.0 48.4 7.0
> 3. 68.85.149.113 0.0% 10 36.3 31.1 22.8 37.2 4.2
> 4. 68.85.243.229 0.0% 10 35.1 27.4 22.3 35.1 4.8
> 5. te-1-3-0-4-cr01.sacramento.c 0.0% 10 35.9 41.1 35.9 50.0 5.9
> 6. pos-0-8-0-0-cr01.sanjose.ca. 0.0% 10 40.2 41.0 38.1 51.3 4.0
> 7. 68.86.87.2 0.0% 10 69.2 44.0 38.9 69.2 9.2
> 8. as15169-1.529bryant.ca.ibone 0.0% 10 55.9 51.0 40.2 85.0 13.2
> 9. 216.239.49.170 0.0% 10 43.0 49.2 41.5 69.8 9.2
> 10. 216.239.49.198 0.0% 10 55.8 57.2 55.0 69.0 4.3
> 11. 216.239.48.34 0.0% 10 59.5 58.8 57.1 61.0 1.5
> 12. 64.233.174.125 0.0% 10 60.2 59.7 58.4 62.3 1.2
> 13. 216.239.46.6 0.0% 10 63.5 66.6 59.7 72.8 4.4
|
Reporter | |
Comment 4•14 years ago
|
||
(In reply to comment #1) > (In reply to comment #0) > > Did anything change about 2 or 3 weeks ago? > > Like the whole network going down? I mean something long lasting, like new hardware or a configuration change. Here's my output of oremj's commands: > clouserw@chiave:~$ mtr --report boris.mozilla.com > HOST: chiave Loss% Snt Last Avg Best Wrst StDev > 1. millhaven 0.0% 10 0.4 0.4 0.4 0.5 0.0 > 2. ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 > 3. ge-7-43-ur01.troutdale.or.bv 0.0% 10 8.6 10.8 8.6 19.2 3.2 > 4. te-3-1-ar01.troutdale.or.bve 0.0% 10 10.0 11.0 7.6 19.9 3.4 > 5. te-1-0-0-0-cr01.seattle.wa.i 0.0% 10 14.6 15.9 12.8 21.4 2.9 > 6. TenGigabitethernet1-4.ar5.SE 0.0% 10 12.7 38.7 12.7 125.5 42.9 > 7. MOZILLA-2.GigabitEthernet4-1 20.0% 10 31.0 30.9 28.2 40.5 4.0 > 8. v9.core2.sj.mozilla.com 30.0% 10 30.6 32.9 27.6 41.7 5.5 > 9. v2.core3.sj.mozilla.com 30.0% 10 29.4 31.0 29.4 32.6 1.2 > 10. boris.mozilla.com 0.0% 10 28.9 29.9 28.4 32.1 1.1 > clouserw@chiave:~$ mtr --report google.com | perl -pne 's/^/> /' > HOST: chiave Loss% Snt Last Avg Best Wrst StDev > 1. millhaven 0.0% 10 0.4 0.4 0.3 0.5 0.0 > 2. ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 > 3. ge-7-43-ur01.troutdale.or.bv 0.0% 10 10.8 12.2 8.6 24.7 5.4 > 4. te-9-3-ar01.troutdale.or.bve 0.0% 10 8.2 14.1 7.9 32.2 8.8 > 5. te-1-0-0-1-cr01.seattle.wa.i 0.0% 10 18.6 15.6 11.7 22.3 3.2 > 6. pos-0-2-0-0-pe01.seattle.wa. 0.0% 10 15.5 16.7 12.7 28.6 4.5 > 7. 75.149.231.90 0.0% 10 32.5 36.4 30.2 59.3 8.8 > 8. 209.85.249.34 0.0% 10 30.4 38.5 30.2 77.4 14.4 > 9. 209.85.250.126 0.0% 10 32.6 36.1 28.7 58.2 9.5 > 10. 64.233.174.97 0.0% 10 41.3 40.2 32.2 55.7 7.3 > 11. 64.233.174.125 50.0% 10 47.8 40.2 32.6 48.6 7.5 > 12. pv-in-f103.1e100.net 40.0% 10 36.0 38.5 34.1 54.5 7.9
|
Assignee | |
Updated•14 years ago
|
Assignee: server-ops → dmoore
|
|
Assignee | |
Comment 6•14 years ago
|
||
We have been tracking ongoing performance problems with Comcast (or our connectivity to Comcast) over the past two weeks. I'd appreciate the contribution of traceroute/mtr outputs from anyone else who has been experiencing unusual delays over the past week or two.
|
||
Comment 7•14 years ago
|
||
Bumping priority. This is affecting a lot of people internally, and who knows how many others trying to download nightly builds, etc.
Severity: normal → major
|
||
Comment 8•14 years ago
|
||
zamboni:~% sudo mtr --report boris.mozilla.com Password: HOST: zamboni Loss% Snt Last Avg Best Wrst StDev 1. mygateway1.motorola 0.0% 10 1.2 1.6 1.1 3.5 0.7 2. ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 3. te-5-1-ur09.sanjose.ca.sfba. 0.0% 10 11.3 10.9 8.4 20.9 3.7 4. 68.85.155.42 0.0% 10 11.8 12.5 10.5 15.7 1.5 5. pos-1-8-0-0-cr01.sanjose.ca. 0.0% 10 17.4 17.3 14.0 32.5 5.4 6. xe-10-3-0.edge1.sanjose1.lev 0.0% 10 15.6 25.8 14.5 106.3 28.4 7. ae-2-79.edge8.sanjose1.level 0.0% 10 103.7 25.0 14.8 103.7 27.7 8. cwie-llc.edge8.sanjose1.leve 20.0% 10 16.8 100.3 15.5 269.7 103.7 9. v8.core2.sj.mozilla.com 60.0% 10 20.6 19.2 16.8 20.6 1.7 10. v2.core3.sj.mozilla.com 30.0% 10 20.9 17.8 15.8 20.9 1.7 11. boris.mozilla.com 0.0% 10 20.2 18.0 15.8 20.2 1.5
Severity: major → normal
damn bugzilla
|
|
||
Comment 11•14 years ago
|
||
$ sudo mtr --report boris.mozilla.com HOST: venona Loss% Snt Last Avg Best Wrst StDev 1. 10.0.1.1 0.0% 10 1.6 2.5 1.3 5.1 1.2 2. ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 3. te-7-1-ur02.sfpine.ca.sfba.c 0.0% 10 11.6 11.0 8.9 12.1 1.1 4. te-0-3-0-7-ar01.oakland.ca.s 0.0% 10 9.6 11.2 8.8 13.9 1.8 5. pos-0-5-0-0-cr01.sacramento. 0.0% 10 12.8 13.8 11.8 17.1 1.8 6. pos-0-7-0-0-cr01.sanjose.ca. 0.0% 10 13.9 13.9 12.4 15.5 0.9 7. xe-7-1-0.edge1.sanjose1.leve 0.0% 10 13.9 13.2 11.7 15.0 1.2 8. ae-3-89.edge8.sanjose1.level 0.0% 10 58.0 18.7 12.6 58.0 13.9 9. cwie-llc.edge8.sanjose1.leve 0.0% 10 13.2 15.6 12.4 25.0 3.7 10. v8.core1.sj.mozilla.com 0.0% 10 16.5 14.6 12.5 16.5 1.4 11. v2.core3.sj.mozilla.com 0.0% 10 14.5 15.1 13.8 16.7 1.1 12. boris.mozilla.com 0.0% 10 15.9 14.7 13.3 15.9 1.0
|
|
||
Comment 12•14 years ago
|
||
|
|
||
Comment 13•14 years ago
|
||
|
|
Assignee | |
Comment 14•14 years ago
|
||
Thanks, everyone. We're continuing to work with Comcast (and Level3, the intermediary provider) to address this problem. I'll try to get another update here shortly.
|
|
Assignee | |
Comment 15•14 years ago
|
||
Rolling this into the existing Comcast bug
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → DUPLICATE
|
||
Updated•14 years ago
|
Component: Server Operations → Server Operations: Netops
|
||
Updated•11 years ago
|
Product: mozilla.org → Infrastructure & Operations
|
||
Updated•2 years ago
|
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•