Closed Bug 571922 Opened 16 years ago Closed 16 years ago

Terribly slow performance connecting to *.mozilla.org

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: robarnold, Assigned: dmoore)

References

Details

(Whiteboard: [gblx])

My update for Firefox is going at no faster than 5KB/sec and bugzilla and mxr are noticeably slower to load as well. Other sites work just fine (I can watch 1080p Youtube videos).
Rob - can you send a traceroute (or an mtr) to 63.245.209.10?
Assignee: server-ops → mrz
[rob@fickle ~]$ traceroute 63.245.209.10 traceroute to 63.245.209.10 (63.245.209.10), 64 hops max, 40 byte packets 1 192.168.166.124 (192.168.166.124) 1.833 ms 1.349 ms 1.262 ms 2 96.179.232.1 (96.179.232.1) 9.120 ms 8.539 ms 9.824 ms 3 68.85.234.1 (68.85.234.1) 10.554 ms 9.013 ms 9.975 ms 4 te-2-1-ar01.pittsburgh.pa.pitt.comcast.net (68.86.100.85) 9.910 ms 12.980 ms 10.137 ms 5 68.85.75.145 (68.85.75.145) 10.477 ms 9.271 ms 9.728 ms 6 te-2-0-0-1-cr01.chicago.il.ibone.comcast.net (68.86.90.177) 23.825 ms 22.238 ms 21.609 ms 7 xe-9-0-0.edge1.Chicago2.Level3.net (4.71.248.33) 21.862 ms 20.811 ms 32.106 ms 8 vlan51.ebr1.Chicago2.Level3.net (4.69.138.158) 21.559 ms 22.587 ms 21.106 ms 9 ae-3-3.ebr2.Denver1.Level3.net (4.69.132.61) 59.014 ms 54.056 ms 53.903 ms 10 ae-1-100.ebr1.Denver1.Level3.net (4.69.132.37) 46.570 ms 53.502 ms 55.121 ms 11 ae-3-3.ebr2.SanJose1.Level3.net (4.69.132.57) 86.841 ms 89.237 ms 89.243 ms 12 ae-82-82.csw3.SanJose1.Level3.net (4.69.134.218) 92.201 ms 90.682 ms 90.126 ms 13 ae-3-89.edge8.SanJose1.Level3.net (4.68.18.148) 82.007 ms ae-4-99.edge8.SanJose1.Level3.net (4.68.18.212) 81.824 ms 81.653 ms 14 CWIE-LLC.edge8.SanJose1.Level3.net (4.53.30.86) 82.299 ms 83.446 ms 92.602 ms 15 v9.core2.sj.mozilla.com (63.245.208.58) 85.656 ms 83.756 ms 82.241 ms 16 * moz.com01.nslb.sj.mozilla.com (63.245.209.10) 82.422 ms 84.100 ms
What's your src addr? http://whatismyipaddress.com/
71.206.238.235
Watching performance right now. Nothing obvious right now (no packet loss). Path is mostly through comcast.net.
You don't see any packet loss do you? If you run mtr for a while and pastebin it, anything?
(In reply to comment #6) > You don't see any packet loss do you? If you run mtr for a while and pastebin > it, anything? I did have my update download quit on me but I didn't see any aborted connections or partially loaded pages. I don't have mtr or a machine where I can run it (it needs all these GUI libraries that my server doesn't/shouldn't have. I could set up ping to monitor packet loss - would that be equivalent?
mtr --curses will skip all the gui.
Building it with WITHOUT_X11=yes seemed to do the trick of not pulling in the GUI libraries. Host Loss% Snt Last Avg Best Wrst StDev 1. 192.168.166.124 0.0% 394 0.9 1.0 0.8 29.0 1.5 2. 96.179.232.1 0.8% 394 13.4 10.6 7.1 40.9 4.4 3. 68.85.234.1 0.0% 394 9.6 10.9 7.4 47.3 4.7 4. te-2-1-ar01.pittsburgh.pa.pitt.comcast.net 0.0% 393 8.7 10.8 7.2 38.8 4.6 5. 68.85.75.145 0.0% 393 8.7 11.0 7.5 38.1 4.9 6. te-2-0-0-1-cr01.chicago.il.ibone.comcast.net 0.0% 393 21.4 24.0 20.4 49.3 4.1 7. xe-9-0-0.edge1.Chicago2.Level3.net 1.0% 393 21.1 28.7 20.0 153.5 15.4 8. vlan51.ebr1.Chicago2.Level3.net 4.6% 393 21.1 23.4 20.1 40.6 3.6 9. ae-3-3.ebr2.Denver1.Level3.net 1.0% 393 45.6 53.2 44.7 74.7 6.1 10. ae-1-100.ebr1.Denver1.Level3.net 2.5% 393 51.7 52.9 44.6 78.8 6.2 11. ae-3-3.ebr2.SanJose1.Level3.net 0.0% 393 91.8 88.9 80.7 116.4 5.8 12. ae-92-92.csw4.SanJose1.Level3.net 0.3% 393 82.4 89.0 80.7 117.1 5.7 13. ae-4-99.edge8.SanJose1.Level3.net 2.0% 393 81.5 86.3 80.7 149.1 10.0 14. CWIE-LLC.edge8.SanJose1.Level3.net 16.0% 393 81.6 87.8 81.1 280.7 18.9 15. v9.core2.sj.mozilla.com 9.4% 393 83.0 84.4 80.9 110.5 4.3 16. moz.com01.nslb.sj.mozilla.com 0.0% 393 83.4 84.3 80.8 109.8 4.7 Number 14 seems consistently bad.
You're the second person to note issues with comcast on the far side. Copying ormej.
This is starting to look a little better for me. CWIE-LLC.edge8.SanJose1.Level3.net and xe-10-2-0.edge1.SanJose1.Level3.net still take 190ms randomly, but I'm not seeing any packet loss atm.
mtr is still showing ~11% packet loss for CWIE-LLC.edge8.SanJose1.Level3.net but my connections to mozilla.org sites appear to be back to reasonable speeds.
I just dropped the GBLX peer. Check again in 5-10 mins (give the route enough time to be withdrawn globally) and let me know what you see.
I'm still getting a terrible download rate for the nightly update. It had improved to 6-7 KB/sec for a minute but has now regressed to the 3-5K/sec. The route from me to the server is the same, but now v9.core2.js.mozilla.com is at 22% loss and CWIE-LLC.edge8.SanJose1.Level3.net is at a 30% loss.
CWIE-LLC.edge8.SanJose1.Level3.net is Mozilla's router and I would worry less about that router than the end point. ICMP is often a lower priority. To 63.245.209.10 or to ftp.mozilla.org, are you seeing packet loss?
Well I tried to download a large file from ftp.mozilla.org and it was going slowly and yeah there was some packet loss. Partway through it sped up 100x and now I can't see the slow speeds anymore. Unfortunately Wireshark hung so I don't have any hard data to share.
I'm seeing the same sort of issues and I am using Comcast on my end. This has been the case for most of the last week.
Assignee: mrz → dmoore
Comcast had issues earlier this week, I'm not seeing problems from my Comcast house. Anyone else?
I was getting 3 KB/sec just 40 minutes ago downloading an update again.
Still seeing packetloss too on: CWIE-LLC.edge8.SanJose1.Level3.net v8.core2.sj.mozilla.com v2.core3.sj.mozilla.com
Had some issues with GBLX that were (finally) resolved late Thursday. Curious if anyone is still seeing problems (and if so, please let me know what your IP address is).
Seems to be fine for the update I just grabbed.
I'm still having prblems with downloads. IP Information: 71.226.9.179 ISP: Comcast Cable Microsoft Windows XP [Version 5.1.2600] (C) Copyright 1985-2001 Microsoft Corp. C:\Documents and Settings\Owner>tracert ftp.mozilla.org Tracing route to dm-ftp01.mozilla.org [63.245.208.138] over a maximum of 30 hops: 1 General failure. Trace complete.
I am now seeing 6-8k/sec as I download an update to the Firefox nightly. Is there anything I can do to help troubleshoot this?
(In reply to comment #23) > Tracing route to dm-ftp01.mozilla.org [63.245.208.138] > 1 General failure. You seem to be blocking outgoing ICMP packets or something, firewall?
(In reply to comment #25) > (In reply to comment #23) > > Tracing route to dm-ftp01.mozilla.org [63.245.208.138] > > 1 General failure. > > You seem to be blocking outgoing ICMP packets or something, firewall? Yes, I found a problem with my firewall rules. I loaded a backup from about a month ago and don't seem to have the same slow download speed. I need to figure out which rule(s)are causing the problem. I was having MD5 checksum errors with MalwareBytes back then and had to setup a new set of rules. Dows this look okay? C:\Documents and Settings\Owner>tracert ftp.mozilla.org Tracing route to dm-ftp01.mozilla.org [63.245.208.138] over a maximum of 30 hops: 1 General failure. Trace complete. C:\Documents and Settings\Owner>tracert ftp.mozilla.org Tracing route to dm-ftp01.mozilla.org [63.245.208.138] over a maximum of 30 hops: 1 5 ms 5 ms 5 ms 71.226.8.1 2 8 ms 7 ms 5 ms 68.85.122.177 3 7 ms 7 ms 7 ms te-9-1-ur01.wndsrsprng.ga.augusta.comcast.net [6 8.86.138.73] 4 8 ms 7 ms 8 ms te-8-4-ar01.augusta.ga.augusta.comcast.net [68.8 6.138.29] 5 8 ms 7 ms 7 ms po-91-ar02.augusta.ga.augusta.comcast.net [68.86 .138.2] 6 7 ms 8 ms 7 ms ge-6-0-0-ar03.augusta.ga.augusta.comcast.net [68 .86.138.93] 7 11 ms 11 ms 11 ms pos-0-11-0-0-cr01.charlotte.nc.ibone.comcast.net [68.86.91.237] 8 18 ms 17 ms 17 ms pos-3-14-0-0-cr01.atlanta.ga.ibone.comcast.net [ 68.86.85.213] 9 17 ms 15 ms 15 ms te-4-4.car1.Atlanta2.Level3.net [4.71.252.13] 10 17 ms 17 ms 17 ms ae-73-52.ebr3.Atlanta2.Level3.net [4.68.103.62] 11 45 ms 37 ms * ae-7.ebr3.Dallas1.Level3.net [4.69.134.21] 12 83 ms 71 ms 71 ms ae-3-3.ebr2.LosAngeles1.Level3.net [4.69.132.77] 13 87 ms 89 ms 89 ms ae-2-2.ebr2.SanJose1.Level3.net [4.69.132.13] 14 92 ms 89 ms 89 ms ae-62-62.csw1.SanJose1.Level3.net [4.69.134.210] 15 89 ms 81 ms 79 ms ae-1-69.edge8.SanJose1.Level3.net [4.68.18.20] 16 82 ms 81 ms 81 ms CWIE-LLC.edge8.SanJose1.Level3.net [4.53.30.86] 17 81 ms 82 ms 81 ms v9.core2.sj.mozilla.com [63.245.208.58] 18 * * * Request timed out. 19 * * * Request timed out. 20 * * * Request timed out. 21 * * * Request timed out. 22 * * * Request timed out. 23 * * * Request timed out. 24 * * * Request timed out. 25 * * * Request timed out. 26 * * * Request timed out. 27 * * * Request timed out. 28 * * * Request timed out. 29 * * * Request timed out. 30 * * * Request timed out. Trace complete.
Well, it must a been a fluke. I tried downloading again with the same backed up firewall ruleset and download speed went to **** again. I was getting around 5.6 KB/sec which is what I've been getting for a couple of weeks now. I never had a problem in the past until recently. I don't have any problems downloading from any other non-mozilla site and my speedtests are fine.
Thanks, everyone. We're continuing to work with Comcast (and Level3, the intermediary provider) to address this problem. I'll try to get another update here shortly.
OS: Windows 7 → All
Hardware: x86 → All
Downloads for me are coming in at approximately 56k modem speeds. http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/ won't fully load in a browser here even after more than 10 minutes' wait. Downloads from kernel.org and gwdg.de don't give me this trouble. I can download a 4G+ distro DVD iso from them faster than a nightly SM build from mozilla.org. I'm on Bright House, not Comcast.
It's not just Comcast. For me it's pretty slow all the time (between 5KBps and 7.5KBps). My connection is 640Kbps (80KBps) cable, from Argentina. Before this problem showed up, I used to download at about 72KBps (normal drop from the theoretical max). My current ip is 190.245.248.212 and this is my winmtr log after a few seconds running. Host - % | Sent | Recv | Best | Avrg | Wrst | Last | -----------------------------|------|------|------|------|------|------| 172.17.0.1 - 0 | 36 | 36 | 0 | 0 | 0 | 0 | No response from host - 100 | 36 | 0 | 0 | 0 | 0 | 0 | No response from host - 100 | 36 | 0 | 0 | 0 | 0 | 0 | No response from host - 100 | 36 | 0 | 0 | 0 | 0 | 0 | No response from host - 100 | 36 | 0 | 0 | 0 | 0 | 0 | 200.89.165.69 - 3 | 36 | 35 | 0 | 15 | 63 | 31 | 200.42.42.69 - 0 | 36 | 36 | 0 | 14 | 31 | 15 | 200.42.42.113 - 3 | 36 | 35 | 0 | 21 | 188 | 188 | 200.42.42.125 - 3 | 35 | 34 | 15 | 18 | 62 | 15 | 64.214.130.253 - 3 | 35 | 34 | 15 | 20 | 109 | 16 | 206.57.3.82 - 32 | 35 | 24 | 156 | 198 | 469 | 156 | 63.245.208.50 - 18 | 35 | 29 | 156 | 177 | 360 | 156 | No response from host - 100 | 35 | 0 | 0 | 0 | 0 | 0 |
Everyone, We've isolated the problem to an upstream carrier and done our best to circumvent the bottleneck. Please compare performance now and update here (whether or not you see an improvement).
(In reply to comment #32) > We've isolated the problem to an upstream carrier and done our best to > circumvent the bottleneck. Exactly when was that done? Last night it seemed to be working normally.
Felix, The fix was made permanent around 10:00 PDT this morning, but last night could have simply been a lull in traffic which relieved the congestion within our the upstream network.
I did see an improvement this afternoon compared to this morning. This morning was about 8.5 - 12.5 KB/sec This afternoon was about 600 KB/sec. Still slow compared to the 1.6 MB/sec I used to get but certainly better. We'll see what happens tomorrow.
zamboni:~% sudo mtr --report boris.mozilla.com Password: HOST: zamboni Loss% Snt Last Avg Best Wrst StDev 1. mygateway1.motorola 0.0% 10 6.1 3.0 1.1 6.1 1.8 2. ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 3. te-5-1-ur09.sanjose.ca.sfba. 0.0% 10 14.4 12.1 9.0 18.9 3.6 4. 68.85.155.42 0.0% 10 11.8 15.0 10.9 22.8 3.5 5. pos-1-8-0-0-cr01.sanjose.ca. 0.0% 10 17.5 19.5 14.2 33.4 6.1 6. xe-10-3-0.edge1.sanjose1.lev 0.0% 10 16.9 22.8 14.3 52.3 14.9 7. ae-2-79.edge8.sanjose1.level 0.0% 10 15.3 16.1 14.6 18.6 1.2 8. cwie-llc.edge8.sanjose1.leve 0.0% 10 23.6 21.0 15.4 25.2 3.0 9. v8.core2.sj.mozilla.com 0.0% 10 37.7 25.0 14.5 50.3 11.1 10. v2.core3.sj.mozilla.com 0.0% 10 31.5 22.2 16.6 31.5 4.5 11. boris.mozilla.com 0.0% 10 20.5 19.1 16.3 23.3 2.4 Haven't downloaded anything from mozilla.com today, but here's my stats.
(In reply to comment #35) > I did see an improvement this afternoon compared to this morning. > > This morning was about 8.5 - 12.5 KB/sec > This afternoon was about 600 KB/sec. > > Still slow compared to the 1.6 MB/sec I used to get but certainly better. We'll > see what happens tomorrow. Tried again just now and topped out at 1.6 MB/sec. :)
Didn't try mtr again, but download speed seems to be ok now. Thanks
Speeds seem fine to me as well. I'm not sure if they were as fast as before but the 500KB/sec I'm getting is fast enough.
GBLX issues are apparently resolved. Turning one the links back online.
Whiteboard: [gblx]
Calling this done.
Status: NEW → RESOLVED
Closed: 16 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.