Closed Bug 571922 Opened 10 years ago Closed 9 years ago

Terribly slow performance connecting to *.mozilla.org

Categories

(mozilla.org Graveyard :: Server Operations, task)

task
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: robarnold, Assigned: dmoore)

References

Details

(Whiteboard: [gblx])

My update for Firefox is going at no faster than 5KB/sec and bugzilla and mxr are noticeably slower to load as well. Other sites work just fine (I can watch 1080p Youtube videos).
Rob - can you send a traceroute (or an mtr) to 63.245.209.10?
Assignee: server-ops → mrz
[rob@fickle ~]$ traceroute 63.245.209.10
traceroute to 63.245.209.10 (63.245.209.10), 64 hops max, 40 byte packets
 1  192.168.166.124 (192.168.166.124)  1.833 ms  1.349 ms  1.262 ms
 2  96.179.232.1 (96.179.232.1)  9.120 ms  8.539 ms  9.824 ms
 3  68.85.234.1 (68.85.234.1)  10.554 ms  9.013 ms  9.975 ms
 4  te-2-1-ar01.pittsburgh.pa.pitt.comcast.net (68.86.100.85)  9.910 ms  12.980 ms  10.137 ms
 5  68.85.75.145 (68.85.75.145)  10.477 ms  9.271 ms  9.728 ms
 6  te-2-0-0-1-cr01.chicago.il.ibone.comcast.net (68.86.90.177)  23.825 ms  22.238 ms  21.609 ms
 7  xe-9-0-0.edge1.Chicago2.Level3.net (4.71.248.33)  21.862 ms  20.811 ms  32.106 ms
 8  vlan51.ebr1.Chicago2.Level3.net (4.69.138.158)  21.559 ms  22.587 ms  21.106 ms
 9  ae-3-3.ebr2.Denver1.Level3.net (4.69.132.61)  59.014 ms  54.056 ms  53.903 ms
10  ae-1-100.ebr1.Denver1.Level3.net (4.69.132.37)  46.570 ms  53.502 ms  55.121 ms
11  ae-3-3.ebr2.SanJose1.Level3.net (4.69.132.57)  86.841 ms  89.237 ms  89.243 ms
12  ae-82-82.csw3.SanJose1.Level3.net (4.69.134.218)  92.201 ms  90.682 ms  90.126 ms
13  ae-3-89.edge8.SanJose1.Level3.net (4.68.18.148)  82.007 ms
    ae-4-99.edge8.SanJose1.Level3.net (4.68.18.212)  81.824 ms  81.653 ms
14  CWIE-LLC.edge8.SanJose1.Level3.net (4.53.30.86)  82.299 ms  83.446 ms  92.602 ms
15  v9.core2.sj.mozilla.com (63.245.208.58)  85.656 ms  83.756 ms  82.241 ms
16  * moz.com01.nslb.sj.mozilla.com (63.245.209.10)  82.422 ms  84.100 ms
What's your src addr?  http://whatismyipaddress.com/
71.206.238.235
Watching performance right now.  Nothing obvious right now (no packet loss).  Path is mostly through comcast.net.
You don't see any packet loss do you?  If you run mtr for a while and pastebin it, anything?
(In reply to comment #6)
> You don't see any packet loss do you?  If you run mtr for a while and pastebin
> it, anything?

I did have my update download quit on me but I didn't see any aborted connections or partially loaded pages. I don't have mtr or a machine where I can run it (it needs all these GUI libraries that my server doesn't/shouldn't have. I could set up ping to monitor packet loss - would that be equivalent?
mtr --curses will skip all the gui.
Building it with WITHOUT_X11=yes seemed to do the trick of not pulling in the GUI libraries.

 Host                                                                   Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 192.168.166.124                                                      0.0%   394    0.9   1.0   0.8  29.0   1.5
 2. 96.179.232.1                                                         0.8%   394   13.4  10.6   7.1  40.9   4.4
 3. 68.85.234.1                                                          0.0%   394    9.6  10.9   7.4  47.3   4.7
 4. te-2-1-ar01.pittsburgh.pa.pitt.comcast.net                           0.0%   393    8.7  10.8   7.2  38.8   4.6
 5. 68.85.75.145                                                         0.0%   393    8.7  11.0   7.5  38.1   4.9
 6. te-2-0-0-1-cr01.chicago.il.ibone.comcast.net                         0.0%   393   21.4  24.0  20.4  49.3   4.1
 7. xe-9-0-0.edge1.Chicago2.Level3.net                                   1.0%   393   21.1  28.7  20.0 153.5  15.4
 8. vlan51.ebr1.Chicago2.Level3.net                                      4.6%   393   21.1  23.4  20.1  40.6   3.6
 9. ae-3-3.ebr2.Denver1.Level3.net                                       1.0%   393   45.6  53.2  44.7  74.7   6.1
10. ae-1-100.ebr1.Denver1.Level3.net                                     2.5%   393   51.7  52.9  44.6  78.8   6.2
11. ae-3-3.ebr2.SanJose1.Level3.net                                      0.0%   393   91.8  88.9  80.7 116.4   5.8
12. ae-92-92.csw4.SanJose1.Level3.net                                    0.3%   393   82.4  89.0  80.7 117.1   5.7
13. ae-4-99.edge8.SanJose1.Level3.net                                    2.0%   393   81.5  86.3  80.7 149.1  10.0
14. CWIE-LLC.edge8.SanJose1.Level3.net                                  16.0%   393   81.6  87.8  81.1 280.7  18.9
15. v9.core2.sj.mozilla.com                                              9.4%   393   83.0  84.4  80.9 110.5   4.3
16. moz.com01.nslb.sj.mozilla.com                                        0.0%   393   83.4  84.3  80.8 109.8   4.7

Number 14 seems consistently bad.
You're the second person to note issues with comcast on the far side.  Copying ormej.
This is starting to look a little better for me.

CWIE-LLC.edge8.SanJose1.Level3.net and xe-10-2-0.edge1.SanJose1.Level3.net still take 190ms randomly, but I'm not seeing any packet loss atm.
mtr is still showing ~11% packet loss for CWIE-LLC.edge8.SanJose1.Level3.net but my connections to mozilla.org sites appear to be back to reasonable speeds.
I just dropped the GBLX peer.  Check again in 5-10 mins (give the route enough time to be withdrawn globally) and let me know what you see.
I'm still getting a terrible download rate for the nightly update. It had improved to 6-7 KB/sec for a minute but has now regressed to the 3-5K/sec. The route from me to the server is the same, but now v9.core2.js.mozilla.com is at 22% loss and CWIE-LLC.edge8.SanJose1.Level3.net is at a 30% loss.
CWIE-LLC.edge8.SanJose1.Level3.net is Mozilla's router and I would worry less about that router than the end point.  ICMP is often a lower priority.

To 63.245.209.10 or to ftp.mozilla.org, are you seeing packet loss?
Well I tried to download a large file from ftp.mozilla.org and it was going slowly and yeah there was some packet loss. Partway through it sped up 100x and now I can't see the slow speeds anymore. Unfortunately Wireshark hung so I don't have any hard data to share.
I'm seeing the same sort of issues and I am using Comcast on my end. This has been the case for most of the last week.
Assignee: mrz → dmoore
Comcast had issues earlier this week, I'm not seeing problems from my Comcast house.

Anyone else?
I was getting 3 KB/sec just 40 minutes ago downloading an update again.
Still seeing packetloss too on:

CWIE-LLC.edge8.SanJose1.Level3.net
v8.core2.sj.mozilla.com
v2.core3.sj.mozilla.com
Had some issues with GBLX that were (finally) resolved late Thursday.  Curious if anyone is still seeing problems (and if so, please let me know what your IP address is).
Seems to be fine for the update I just grabbed.
I'm still having prblems with downloads.

IP Information: 71.226.9.179
ISP: Comcast Cable

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Owner>tracert ftp.mozilla.org

Tracing route to dm-ftp01.mozilla.org [63.245.208.138]
over a maximum of 30 hops:

  1  General failure.

Trace complete.
I am now seeing 6-8k/sec as I download an update to the Firefox nightly. Is there anything I can do to help troubleshoot this?
(In reply to comment #23)
> Tracing route to dm-ftp01.mozilla.org [63.245.208.138]
>   1  General failure.

You seem to be blocking outgoing ICMP packets or something, firewall?
(In reply to comment #25)
> (In reply to comment #23)
> > Tracing route to dm-ftp01.mozilla.org [63.245.208.138]
> >   1  General failure.
> 
> You seem to be blocking outgoing ICMP packets or something, firewall?

Yes, I found a problem with my firewall rules. I loaded a backup from about a month ago and don't seem to have the same slow download speed. I need to figure out which rule(s)are causing the problem.

I was having MD5 checksum errors with MalwareBytes back then and had to setup a new set of rules.

Dows this look okay?

C:\Documents and Settings\Owner>tracert ftp.mozilla.org

Tracing route to dm-ftp01.mozilla.org [63.245.208.138]
over a maximum of 30 hops:

  1  General failure.

Trace complete.

C:\Documents and Settings\Owner>tracert ftp.mozilla.org

Tracing route to dm-ftp01.mozilla.org [63.245.208.138]
over a maximum of 30 hops:

  1     5 ms     5 ms     5 ms  71.226.8.1
  2     8 ms     7 ms     5 ms  68.85.122.177
  3     7 ms     7 ms     7 ms  te-9-1-ur01.wndsrsprng.ga.augusta.comcast.net [6
8.86.138.73]
  4     8 ms     7 ms     8 ms  te-8-4-ar01.augusta.ga.augusta.comcast.net [68.8
6.138.29]
  5     8 ms     7 ms     7 ms  po-91-ar02.augusta.ga.augusta.comcast.net [68.86
.138.2]
  6     7 ms     8 ms     7 ms  ge-6-0-0-ar03.augusta.ga.augusta.comcast.net [68
.86.138.93]
  7    11 ms    11 ms    11 ms  pos-0-11-0-0-cr01.charlotte.nc.ibone.comcast.net
 [68.86.91.237]
  8    18 ms    17 ms    17 ms  pos-3-14-0-0-cr01.atlanta.ga.ibone.comcast.net [
68.86.85.213]
  9    17 ms    15 ms    15 ms  te-4-4.car1.Atlanta2.Level3.net [4.71.252.13]
 10    17 ms    17 ms    17 ms  ae-73-52.ebr3.Atlanta2.Level3.net [4.68.103.62]

 11    45 ms    37 ms     *     ae-7.ebr3.Dallas1.Level3.net [4.69.134.21]
 12    83 ms    71 ms    71 ms  ae-3-3.ebr2.LosAngeles1.Level3.net [4.69.132.77]

 13    87 ms    89 ms    89 ms  ae-2-2.ebr2.SanJose1.Level3.net [4.69.132.13]
 14    92 ms    89 ms    89 ms  ae-62-62.csw1.SanJose1.Level3.net [4.69.134.210]

 15    89 ms    81 ms    79 ms  ae-1-69.edge8.SanJose1.Level3.net [4.68.18.20]
 16    82 ms    81 ms    81 ms  CWIE-LLC.edge8.SanJose1.Level3.net [4.53.30.86]

 17    81 ms    82 ms    81 ms  v9.core2.sj.mozilla.com [63.245.208.58]
 18     *        *        *     Request timed out.
 19     *        *        *     Request timed out.
 20     *        *        *     Request timed out.
 21     *        *        *     Request timed out.
 22     *        *        *     Request timed out.
 23     *        *        *     Request timed out.
 24     *        *        *     Request timed out.
 25     *        *        *     Request timed out.
 26     *        *        *     Request timed out.
 27     *        *        *     Request timed out.
 28     *        *        *     Request timed out.
 29     *        *        *     Request timed out.
 30     *        *        *     Request timed out.

Trace complete.
Well, it must a been a fluke. I tried downloading again with the same backed up firewall ruleset and download speed went to **** again. I was getting around 5.6 KB/sec which is what I've been getting for a couple of weeks now. I never had a problem in the past until recently. 

I don't have any problems downloading from any other non-mozilla site and my speedtests are fine.
Duplicate of this bug: 575325
Thanks, everyone.

We're continuing to work with Comcast (and Level3, the intermediary provider)
to address this problem. I'll try to get another update here shortly.
OS: Windows 7 → All
Hardware: x86 → All
Downloads for me are coming in at approximately 56k modem speeds. http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/ won't fully load in a browser here even after more than 10 minutes' wait. Downloads from kernel.org and gwdg.de don't give me this trouble. I can download a 4G+ distro DVD iso from them faster than a nightly SM build from mozilla.org. I'm on Bright House, not Comcast.
It's not just Comcast.
For me it's pretty slow all the time (between 5KBps and 7.5KBps). My connection is 640Kbps (80KBps) cable, from Argentina. Before this problem showed up, I used to download at about 72KBps (normal drop from the theoretical max).
My current ip is 190.245.248.212 and this is my winmtr log after a few seconds running.

    Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
-----------------------------|------|------|------|------|------|------|
           172.17.0.1 -    0 |   36 |   36 |    0 |    0 |    0 |    0 |
No response from host -  100 |   36 |    0 |    0 |    0 |    0 |    0 |
No response from host -  100 |   36 |    0 |    0 |    0 |    0 |    0 |
No response from host -  100 |   36 |    0 |    0 |    0 |    0 |    0 |
No response from host -  100 |   36 |    0 |    0 |    0 |    0 |    0 |
        200.89.165.69 -    3 |   36 |   35 |    0 |   15 |   63 |   31 |
         200.42.42.69 -    0 |   36 |   36 |    0 |   14 |   31 |   15 |
        200.42.42.113 -    3 |   36 |   35 |    0 |   21 |  188 |  188 |
        200.42.42.125 -    3 |   35 |   34 |   15 |   18 |   62 |   15 |
       64.214.130.253 -    3 |   35 |   34 |   15 |   20 |  109 |   16 |
          206.57.3.82 -   32 |   35 |   24 |  156 |  198 |  469 |  156 |
        63.245.208.50 -   18 |   35 |   29 |  156 |  177 |  360 |  156 |
No response from host -  100 |   35 |    0 |    0 |    0 |    0 |    0 |
Everyone,

We've isolated the problem to an upstream carrier and done our best to circumvent the bottleneck. Please compare performance now and update here (whether or not you see an improvement).
(In reply to comment #32)
> We've isolated the problem to an upstream carrier and done our best to
> circumvent the bottleneck.

Exactly when was that done? Last night it seemed to be working normally.
Felix,

The fix was made permanent around 10:00 PDT this morning, but last night could have simply been a lull in traffic which relieved the congestion within our the upstream network.
I did see an improvement this afternoon compared to this morning.

This morning was about 8.5 - 12.5 KB/sec
This afternoon was about 600 KB/sec.

Still slow compared to the 1.6 MB/sec I used to get but certainly better. We'll see what happens tomorrow.
zamboni:~% sudo mtr --report boris.mozilla.com                                 
Password:
HOST: zamboni                     Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. mygateway1.motorola           0.0%    10    6.1   3.0   1.1   6.1   1.8
  2. ???                          100.0    10    0.0   0.0   0.0   0.0   0.0
  3. te-5-1-ur09.sanjose.ca.sfba.  0.0%    10   14.4  12.1   9.0  18.9   3.6
  4. 68.85.155.42                  0.0%    10   11.8  15.0  10.9  22.8   3.5
  5. pos-1-8-0-0-cr01.sanjose.ca.  0.0%    10   17.5  19.5  14.2  33.4   6.1
  6. xe-10-3-0.edge1.sanjose1.lev  0.0%    10   16.9  22.8  14.3  52.3  14.9
  7. ae-2-79.edge8.sanjose1.level  0.0%    10   15.3  16.1  14.6  18.6   1.2
  8. cwie-llc.edge8.sanjose1.leve  0.0%    10   23.6  21.0  15.4  25.2   3.0
  9. v8.core2.sj.mozilla.com       0.0%    10   37.7  25.0  14.5  50.3  11.1
 10. v2.core3.sj.mozilla.com       0.0%    10   31.5  22.2  16.6  31.5   4.5
 11. boris.mozilla.com             0.0%    10   20.5  19.1  16.3  23.3   2.4

Haven't downloaded anything from mozilla.com today, but here's my stats.
Duplicate of this bug: 575590
(In reply to comment #35)
> I did see an improvement this afternoon compared to this morning.
> 
> This morning was about 8.5 - 12.5 KB/sec
> This afternoon was about 600 KB/sec.
> 
> Still slow compared to the 1.6 MB/sec I used to get but certainly better. We'll
> see what happens tomorrow.

Tried again just now and topped out at 1.6 MB/sec. :)
Didn't try mtr again, but download speed seems to be ok now. Thanks
Speeds seem fine to me as well. I'm not sure if they were as fast as before but the 500KB/sec I'm getting is fast enough.
GBLX issues are apparently resolved.  Turning one the links back online.
Whiteboard: [gblx]
Calling this done.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.