Closed Bug 420221 Opened 16 years ago Closed 16 years ago

Slower Ftp speed on <ftp.eu.mozilla.org> (= <ftp.mozilla.org>) recently

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Platform:
x86
Windows 2000
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: sgautherie, Assigned: mrz)

Details

(Keywords: regression) 

A while back, some days, it happened that I could (FTP) download at like "287" KB/s (AFAICremember), which was just great;
but usually, I get 86 KB/s, steady.

But, for some time now (about 2008.02.15 !?), I'm getting between 50 and 65 KB/s only:
the transfer seems to make repeated brief pauses :-(

I'm usually using FileZilla v2.2.32;
but I checked that Windows |ftp| shows the same slowness.

I'd like to get back to 86 KB/s; getting more would be even better...
(I'm DL'ing at least a SeaMonkey nightly each day; some days, multiple FF/SM/TB files :-<)

NB: I have no issue getting much higher speeds when (HTTP) downloading files from various web sites ... The issue doesn't seem to be on "my" end.
Back in the day, ftp.eu was a round-robin DNS through several EU mirrors.  But that was some time ago.

Right now it's only going to Mozilla's ftp server.  If you goto http://ftp.mozilla.org/ do you get similar transfer rates?
Assignee: server-ops → mrz
1st test:
I confirm that (Ftp) connecting directly to <ftp.mozilla.org> with FileZilla gives the same speed limitation:
I had 3 downloads at the same time, all with frequent pauses and 55-65 KB/s speed.
From my point of view, it's very much like the Ftp server is limiting the speed on a per download/file basis :-(

2nd test:
I tried <http://ftp.mozilla.org/>,
with SeaMonkey v2.0a1pre: 3 files, speed moving between 60-100 KB/s;
with MsIE v6.01: 1 file, steady at 70+ KB/s.
Then, Http would seem a little faster than Ftp, but not that much.
NB: I think this Ftp vs Http difference has "always" been there.

3rd test:
I tried <http://releases.mozilla.org/>,
with SeaMonkey v2.0a1pre: 1 file, steady at 202 KB/s.
Then, it's the ftp "site" which limits the speed: 3-4 times slower.

***

<http://ftp.mozilla.org/> says "High bandwidth servers that contain the public release files are available at releases.mozilla.org.".

Even if the Ftp site has lower bandwith available, it seems it has an even stricter limitation than its available (network) bandwith ? :-(
Summary: Slower <ftp.eu.mozilla.org> recently → Slower Ftp speed on <ftp.eu.mozilla.org> (= <ftp.mozilla.org>) recently
Tested from three seperate Mozilla locations.  I grabbed 

http://ftp.mozilla.org/pub/mozilla.org/seamonkey/nightly/2008-03-02-01-trunk/seamonkey-2.0a1pre.sv-SE.mac.dmg

From a host local to ftp.mozilla.org - 10.1M/sec.

From Amsterdam - 715K/sec.

From China - 85K/sec.

From Toronto, Canada - 100K/sec.

Doing the same with ftp,  ftp://ftp.mozilla.org/pub/mozilla.org/seamonkey/nightly/2008-03-02-01-trunk/seamonkey-2.0a1pre.sv-SE.mac.dmg - 

From a host local to ftp.mozilla.org - 11.2M/sec.

From Amsterdam - 716K/sec.

From China - 203K/sec.

From Toronto, Canada - 82K/sec.

Wasn't able to duplicate any of the pauses or transfer rates you were experiencing. Doesn't appear to be an issue with the origin server but something along your path that I haven't been able to narrow down.  

I hesitate to just close this but it appears to be something between ftp.mozilla.org and you that's causing the problem, probably far outside my reach or ability to fix.

Perhaps sending a traceroute or 'mtr' to ftp.mozilla.org (or 63.245.209.10) would shed some light on it?
{{
>tracert ftp.mozilla.org

Détermination de l'itinéraire vers dm-ftp01.mozilla.org [63.245.208.138] avec un maximum de 30 sauts :

  1     5 ms     5 ms     6 ms  82.234.249.254
  2     6 ms    14 ms     5 ms  78.254.1.62
  3     6 ms     7 ms     6 ms  seg75-1.v902.intf.nra.proxad.net [78.254.255.37]
  4     6 ms     6 ms     6 ms  inv75-1-v900.intf.nra.proxad.net [78.254.255.33]
  5     6 ms     6 ms     6 ms  lit75-1-v902.intf.nra.proxad.net [78.254.255.29]
  6     9 ms     6 ms     6 ms  bne75-1-v900.intf.nra.proxad.net [78.254.255.25]
  7    12 ms     5 ms     6 ms  ras75-1-v902.intf.nra.proxad.net [78.254.255.21]
  8     6 ms     5 ms     6 ms  dan75-1-v900.intf.nra.proxad.net [78.254.255.17]
  9     6 ms     6 ms     7 ms  gob75-1-v902.intf.nra.proxad.net [78.254.255.13]
 10     6 ms     6 ms    10 ms  bob75-1-v900.intf.nra.proxad.net [78.254.255.9]
 11    19 ms     6 ms     6 ms  mna75-1-v902.intf.nra.proxad.net [78.254.255.5]
 12     6 ms     *        6 ms  th2-6k-2-1-po1.intf.nra.proxad.net [78.254.255.1]
 13     *        *        7 ms  bzn-crs16-1-be1004.intf.routers.proxad.net [212.27.50.173]
 14     *        *       14 ms  londres-6k-1-po101.intf.routers.proxad.net [212.27.51.186]
 15    14 ms    14 ms    17 ms  linx.ge1-0.cr01.lhr01.mzima.net [195.66.225.15]
 16   105 ms    92 ms   105 ms  eos1-0.cr01.lga02.mzima.net [216.193.255.45]
 17    97 ms    93 ms   104 ms  xe1-0.cr01.lga01.mzima.net [216.193.255.213]
 18   112 ms   122 ms   113 ms  eos3-2.cr01.ord01.mzima.net [216.193.255.54]
 19   160 ms   160 ms   160 ms  eos1-22.cr02.sjc02.mzima.net [216.193.255.154]
 20   159 ms   160 ms   354 ms  ge1-mozilla.cust.sjc02.mzima.net [72.37.156.166]
 21   159 ms   159 ms   159 ms  v9.core2.sj.mozilla.com [63.245.208.58]
 22     *        *        *     Délai d'attente de la demande dépassé.
 23     *        *        *     Délai d'attente de la demande dépassé.
 24     *        *        *     Délai d'attente de la demande dépassé.
 25     *        *        *     Délai d'attente de la demande dépassé.
 26     *        *        *     Délai d'attente de la demande dépassé.
 27     *        *        *     Délai d'attente de la demande dépassé.
 28     *        *        *     Délai d'attente de la demande dépassé.
 29     *        *        *     Délai d'attente de la demande dépassé.
 30     *        *        *     Délai d'attente de la demande dépassé.

Itinéraire déterminé.


>tracert 63.245.209.10

Détermination de l'itinéraire vers moz.com01.nslb.sj.mozilla.com [63.245.209.10] avec un maximum de 30 sauts :

  1     9 ms     6 ms     5 ms  82.234.249.254
  2     6 ms     5 ms     8 ms  78.254.1.62
  3    14 ms    10 ms    18 ms  seg75-1.v902.intf.nra.proxad.net [78.254.255.37]
  4     5 ms     5 ms     5 ms  inv75-1-v900.intf.nra.proxad.net [78.254.255.33]
  5     5 ms    11 ms    10 ms  lit75-1-v902.intf.nra.proxad.net [78.254.255.29]
  6     5 ms     6 ms     5 ms  bne75-1-v900.intf.nra.proxad.net [78.254.255.25]
  7     6 ms     6 ms     5 ms  ras75-1-v902.intf.nra.proxad.net [78.254.255.21]
  8     6 ms     5 ms     5 ms  dan75-1-v900.intf.nra.proxad.net [78.254.255.17]
  9     6 ms     6 ms     6 ms  gob75-1-v902.intf.nra.proxad.net [78.254.255.13]
 10     6 ms     6 ms     6 ms  bob75-1-v900.intf.nra.proxad.net [78.254.255.9]
 11     6 ms     8 ms     6 ms  mna75-1-v902.intf.nra.proxad.net [78.254.255.5]
 12     *        *        6 ms  th2-6k-2-1-po1.intf.nra.proxad.net [78.254.255.1]
 13     *        *        *     Délai d'attente de la demande dépassé.
 14    13 ms    18 ms     *     londres-6k-1-po101.intf.routers.proxad.net [212.27.51.186]
 15    16 ms    17 ms    17 ms  linx.ge1-0.cr01.lhr01.mzima.net [195.66.225.15]
 16    92 ms    93 ms   103 ms  eos1-0.cr01.lga02.mzima.net [216.193.255.45]
 17    93 ms    92 ms   104 ms  xe1-0.cr01.lga01.mzima.net [216.193.255.213]
 18   111 ms   111 ms   121 ms  eos3-2.cr01.ord01.mzima.net [216.193.255.54]
 19   166 ms   161 ms   161 ms  eos1-22.cr02.sjc02.mzima.net [216.193.255.154]
 20   159 ms   159 ms   159 ms  ge0-mozilla.cust.sjc02.mzima.net [72.37.156.162]
 21   159 ms   161 ms   160 ms  v9.core1.sj.mozilla.com [63.245.208.57]
 22   160 ms   159 ms   159 ms  moz.com01.nslb.sj.mozilla.com [63.245.209.10]

Itinéraire déterminé.
}}

If I interpret it correctly:
*proxad.net (Paris->London) is my ISP network: seems perfectly fine.
*"90ms at eos1-0.cr01.lga02.mzima.net" is to get to the USA, I guess.
*mzima.net (Mozilla.org ISP !?): it jumps to 110ms then 160ms.
 *Is this for going from east coast to west coast ?
 *Is it the lower delay to be expected ?
*mozilla.com (<-> mzima.net) is perfectly fine.

From these figures, the "issue" (if any) seems to be _inside_ the mzima.net network.

Do you want a Windows pathping output to help further ?
To compare with:
{{
>tracert releases.mozilla.org

Détermination de l'itinéraire vers releases.mozilla.org [64.50.238.52]
avec un maximum de 30 sauts :

  1     6 ms     6 ms     5 ms  82.234.249.254
  2     6 ms     5 ms     5 ms  78.254.1.62
  3     6 ms     6 ms     5 ms  seg75-1.v902.intf.nra.proxad.net [78.254.255.37]
  4     5 ms     6 ms     5 ms  inv75-1-v900.intf.nra.proxad.net [78.254.255.33]
  5     6 ms     6 ms     6 ms  lit75-1-v902.intf.nra.proxad.net [78.254.255.29]
  6     6 ms     6 ms     6 ms  bne75-1-v900.intf.nra.proxad.net [78.254.255.25]
  7     6 ms     5 ms     6 ms  ras75-1-v902.intf.nra.proxad.net [78.254.255.21]
  8     6 ms     6 ms     5 ms  dan75-1-v900.intf.nra.proxad.net [78.254.255.17]
  9     6 ms     6 ms     6 ms  gob75-1-v902.intf.nra.proxad.net [78.254.255.13]
 10     6 ms     6 ms     6 ms  bob75-1-v900.intf.nra.proxad.net [78.254.255.9]
 11     7 ms     8 ms     8 ms  mna75-1-v902.intf.nra.proxad.net [78.254.255.5]
 12     *        6 ms     *     th2-6k-2-1-po1.intf.nra.proxad.net [78.254.255.1]
 13     *        *        *     Délai d'attente de la demande dépassé.
 14     8 ms     7 ms     7 ms  th2-crs16-1-be1000.intf.routers.proxad.net [212.27.57.202]
 15     7 ms     7 ms     *     te-3-2.car1.Paris1.Level3.net [212.73.207.13]
 16     8 ms    17 ms    17 ms  ae-32-54.ebr2.Paris1.Level3.net [4.68.109.126]
 17    89 ms    89 ms    91 ms  ae-41.ebr2.Washington1.Level3.net [4.69.137.50]
 18    90 ms    91 ms    90 ms  ae-92-92.csw4.Washington1.Level3.net [4.69.134.158]
 19    96 ms    90 ms    91 ms  ae-94-94.ebr4.Washington1.Level3.net [4.69.134.189]
 20   109 ms   107 ms   107 ms  ae-3.ebr4.NewYork1.Level3.net [4.69.132.94]
 21    95 ms   103 ms    94 ms  ae-84-84.csw3.NewYork1.Level3.net [4.69.134.122]
 22    96 ms    95 ms    96 ms  ae-33-89.car3.NewYork1.Level3.net [4.68.16.133]
 23    95 ms    95 ms    95 ms  TDS-TELECOM.car3.NewYork1.Level3.net [4.71.172.154]
 24   114 ms   112 ms   113 ms  atlngacor01-p0-0.network.tds.net [64.50.239.10]
 25   114 ms   113 ms   113 ms  ftp-atl.osuosl.org [64.50.238.52]
}}

This uses Level3.net instead of mzima.net.
Then it would seem 110ms is the delay (to expect) when not going further than the east coast.

Does this "explain" that the 50-60 KB/s I'm getting is the best I can hope for ?

***

From your figures,
I don't know about China,
but I'm "puzzled" by the "low" speed from Toronto,
and I would very much like to get the _high_ speed of Amsterdam...

Why would Amsterdam be 10 times faster than Paris ?
releases.mozilla.org is a round-robin DNS and not hosted in Mozilla's datacenters.  Your better test would be to 63.245.209.10.  But that said, your traceroute to ftp.mozilla.org looks fine. 

Amsterdam to San Jose is all on Mzima, one of Mozilla's upstream providers.  Toronto has other connectivity issues that make it look poor. 

If you're actually downloading from releases.mozilla.org then yes, that speed probably makes a lot of sense.  
I could succeed 1 pathping only:
{{
>pathping 63.245.209.10

Détermination de l'itinéraire vers moz.com01.nslb.sj.mozilla.com [63.245.209.10]
avec un maximum de 30 sauts :
  0  vau75-7-82-234-249-215.fbx.proxad.net [82.234.249.215]
  1  82.234.249.254
  2  78.254.1.62
  3  seg75-1.v902.intf.nra.proxad.net [78.254.255.37]
  4  inv75-1-v900.intf.nra.proxad.net [78.254.255.33]
  5  lit75-1-v902.intf.nra.proxad.net [78.254.255.29]
  6  bne75-1-v900.intf.nra.proxad.net [78.254.255.25]
  7  ras75-1-v902.intf.nra.proxad.net [78.254.255.21]
  8  dan75-1-v900.intf.nra.proxad.net [78.254.255.17]
  9  gob75-1-v902.intf.nra.proxad.net [78.254.255.13]
 10  bob75-1-v900.intf.nra.proxad.net [78.254.255.9]
 11  mna75-1-v902.intf.nra.proxad.net [78.254.255.5]
 12  .th2-6k-2-1-po1.intf.nra.proxad.net [78.254.255.1]
 13  ..bzn-crs16-1-be1004.intf.routers.proxad.net [212.27.50.173]
 14  londres-6k-1-po101.intf.routers.proxad.net [212.27.51.186]
 15  linx.ge1-0.cr01.lhr01.mzima.net [195.66.225.15]
 16  eos1-0.cr01.lga02.mzima.net [216.193.255.45]
 17  xe1-0.cr01.lga01.mzima.net [216.193.255.213]
 18  eos3-2.cr01.ord01.mzima.net [216.193.255.54]
 19  eos1-22.cr02.sjc02.mzima.net [216.193.255.154]
 20  ge0-mozilla.cust.sjc02.mzima.net [72.37.156.162]
 21  v9.core1.sj.mozilla.com [63.245.208.57]
 22  moz.com01.nslb.sj.mozilla.com [63.245.209.10]

Traitement des statistiques pendant 550 secondes...
            Source vers ici  Ce noud/lien
Saut RTT    Perdu/Envoyé =  Perdu/Envoyé =  Adresse
  0                                           vau75-7-82-234-249-215.fbx.proxad.net [82.234.249.215]
                                0/ 100 =  0%   |
  1    5ms     0/ 100 =  0%     0/ 100 =  0%  82.234.249.254
                                0/ 100 =  0%   |
  2    5ms     0/ 100 =  0%     0/ 100 =  0%  78.254.1.62
                                0/ 100 =  0%   |
  3    6ms     0/ 100 =  0%     0/ 100 =  0%  seg75-1.v902.intf.nra.proxad.net [78.254.255.37]
                                0/ 100 =  0%   |
  4    6ms     0/ 100 =  0%     0/ 100 =  0%  inv75-1-v900.intf.nra.proxad.net [78.254.255.33]
                                0/ 100 =  0%   |
  5    6ms     0/ 100 =  0%     0/ 100 =  0%  lit75-1-v902.intf.nra.proxad.net [78.254.255.29]
                                0/ 100 =  0%   |
  6    9ms     0/ 100 =  0%     0/ 100 =  0%  bne75-1-v900.intf.nra.proxad.net [78.254.255.25]
                                0/ 100 =  0%   |
  7    6ms     0/ 100 =  0%     0/ 100 =  0%  ras75-1-v902.intf.nra.proxad.net [78.254.255.21]
                                0/ 100 =  0%   |
  8    7ms     0/ 100 =  0%     0/ 100 =  0%  dan75-1-v900.intf.nra.proxad.net [78.254.255.17]
                                0/ 100 =  0%   |
  9    8ms     0/ 100 =  0%     0/ 100 =  0%  gob75-1-v902.intf.nra.proxad.net [78.254.255.13]
                                0/ 100 =  0%   |
 10    6ms     0/ 100 =  0%     0/ 100 =  0%  bob75-1-v900.intf.nra.proxad.net [78.254.255.9]
                                0/ 100 =  0%   |
 11    7ms     0/ 100 =  0%     0/ 100 =  0%  mna75-1-v902.intf.nra.proxad.net [78.254.255.5]
                                0/ 100 =  0%   |
 12    6ms     0/ 100 =  0%     0/ 100 =  0%  th2-6k-2-1-po1.intf.nra.proxad.net [78.254.255.1]
                                0/ 100 =  0%   |
 13    8ms     1/ 100 =  1%     1/ 100 =  1%  bzn-crs16-1-be1004.intf.routers.proxad.net [212.27.50.173]
                                0/ 100 =  0%   |
 14   16ms     0/ 100 =  0%     0/ 100 =  0%  londres-6k-1-po101.intf.routers.proxad.net [212.27.51.186]
                                0/ 100 =  0%   |
 15   19ms     0/ 100 =  0%     0/ 100 =  0%  linx.ge1-0.cr01.lhr01.mzima.net [195.66.225.15]
                                0/ 100 =  0%   |
 16   97ms     0/ 100 =  0%     0/ 100 =  0%  eos1-0.cr01.lga02.mzima.net [216.193.255.45]
                                0/ 100 =  0%   |
 17   97ms     0/ 100 =  0%     0/ 100 =  0%  xe1-0.cr01.lga01.mzima.net [216.193.255.213]
                                0/ 100 =  0%   |
 18  115ms     0/ 100 =  0%     0/ 100 =  0%  eos3-2.cr01.ord01.mzima.net [216.193.255.54]
                                0/ 100 =  0%   |
 19  163ms     0/ 100 =  0%     0/ 100 =  0%  eos1-22.cr02.sjc02.mzima.net [216.193.255.154]
                                0/ 100 =  0%   |
 20  170ms    34/ 100 = 34%    34/ 100 = 34%  ge0-mozilla.cust.sjc02.mzima.net [72.37.156.162]
                                0/ 100 =  0%   |
 21  159ms     0/ 100 =  0%     0/ 100 =  0%  v9.core1.sj.mozilla.com [63.245.208.57]
                                0/ 100 =  0%   |
 22  159ms     0/ 100 =  0%     0/ 100 =  0%  moz.com01.nslb.sj.mozilla.com [63.245.209.10]

Itinéraire déterminé.
}}
On this test, |ge0-mozilla.cust.sjc02.mzima.net| lost 34% of the packets :-/
That could be worth investigating !?

***

Ah, succeeded another one:
{{
 13    7ms     0/ 100 =  0%     0/ 100 =  0%  bzn-crs16-1-be1004.intf.routers.proxad.net [212.27.50.173]
                                0/ 100 =  0%   |
 20  163ms    34/ 100 = 34%    34/ 100 = 34%  ge0-mozilla.cust.sjc02.mzima.net [72.37.156.162]
                                0/ 100 =  0%   |
}}

There is something wrong with |ge0-mozilla.cust.sjc02.mzima.net| !
(I guess Amsterdam is not going through this one...)
Just because a router is dropping icmp packets means nothing - most routers give icmp the lowest priority.  We haven't seen any other issue or complaints, and if that router had an issue we certainly would.  Really what matters is if you are seeing any packet loss to moz.com01.nslb.sj.mozilla.com, not an intermediate router, which you aren't.  Furthermore, I go through this same hop from my home, and have perfect connectivity to our colocation.
FTP chooses random port numbers above 10000 for the data channel for your FTP connection.  HTTP is almost always on port 80 or 443.  I would bet what's happening is your ISP is prioritizing HTTP traffic because that's what most people use, and throttling anything on nonstandard ports.  Traffic shaping is pretty common on residential ISPs these days.
I'm inclined to agree with Dave Miller - you appear to be rate-limited somewhere between your and Mozilla.  

btw, ge0-mozilla.cust.sjc02.mzima.net is my router's interface in San Jose so Amsterdam would have to go through it.  But you don't appear to be getting packet loss end-to-end.  

In any event, there's little I can do.  We're working on more .eu FTP mirrors and perhaps that will help.
Status: NEW → RESOLVED
Closed: 16 years ago
Resolution: --- → WONTFIX
1)
My apologies for thinking that dropping ICMP packets implied an issue with actual data transfer.

2)
But, I don't think my ISP is limiting my (Ftp) transfer rate.
{{
>ftp releases.mozilla.org

Connecté à releases.mozilla.org.
220 Welcome to the Mozilla releases FTP server
ftp> cd /pub/mozilla.org/seamonkey/releases/1.1.8/
ftp> get seamonkey-1.1.8.en-US.win32.zip
200 PORT command successful. Consider using PASV.
150 Opening BINARY mode data connection for seamonkey-1.1.8.en-US.win32.zip (11300841 bytes).
226 File send OK.
ftp : 11300841 octets reçus dans 29,47Secondes 383,50Ko/sec.

***

>ftp ftp.mozilla.org

Connecté à dm-ftp01.mozilla.org.
ftp : 11300841 octets reçus dans 194,31Secondes 58,16Ko/sec.
}}

My question is:
Why am now I getting 58 KB/s, when I used to get 86 KB/s ... and when my end can, at least, handle 384 KB/s ?
That's not your (high) 715 KB/s from Amsterdam, but that's not even your (low) 82 KB/s from Toronto either...

I gave it another try
{{
>ftp releases.mozilla.org

Connecté à releases.mozilla.org.
220-Welcome to TDS Internet Services - mirror3.mirrors.tds.net FTP service.
ftp : 11300841 octets reçus dans 23,86Secondes 473,63Ko/sec.
}}

Then, I fully agree that the small pauses I currently see look very much like throttling.
Would there be any mean to find out at which "hop" this throttling occurs ?

3)
'releases.' gives me good speed for the time being (to download releases) :-)
Yes, having at least one 'ftp.eu.' actually located in Europe might help (for nightlies/...).
(but that's so much more data to mirror...)
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.