Open Bug 1910344 Opened 1 year ago Updated 11 months ago

Slow download of source archive with 4.44 MB/s

Categories

(Cloud Services :: Operations: Product Delivery, defect)

defect

Tracking

(Not tracked)

UNCONFIRMED

People

(Reporter: pmenzel+bugzilla.mozilla.org, Unassigned)

References

()

Details

Steps to reproduce:

Download the source archive:

$ wget https://ftp.mozilla.org/pub/firefox/releases/128.0.3/source/firefox-128.0.3.source.tar.xz
[…]
--2024-07-29 11:02:53--  https://ftp.mozilla.org/pub/firefox/releases/128.0.3/source/firefox-128.0.3.source.tar.xz
Resolving ftp.mozilla.org... 34.117.35.28
Connecting to ftp.mozilla.org|34.117.35.28|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 563698520 (538M) [application/octet-stream]
Saving to: '/dev/shm/bee-pmenzel/firefox/files/firefox-128.0.3.source.tar.xz'
[…]
2024-07-29 11:04:55 (4.44 MB/s) - '/dev/shm/bee-pmenzel/firefox/files/firefox-128.0.3.source.tar.xz' saved [563698520/563698520]

$ lsof -i
[…]
wget      70138 pmenzel 4u  IPv4 6821661      0t0  TCP hotelcalifornia.molgen.mpg.de:33976->28.35.117.34.bc.googleusercontent.com:https (ESTABLISHED)

Actual results:

Although we have a 10 Gbit/s connection, the download speed was only 4.44 MB/s.

Expected results:

The speed should have been at least 100 MB/s.

The following server was used:

Group: mozilla-employee-confidential
Component: Untriaged → RelOps: General
Product: Firefox → Infrastructure & Operations
Version: Firefox 130 → Production
Group: mozilla-employee-confidential
Component: RelOps: General → General
Product: Infrastructure & Operations → Release Engineering
QA Contact: jlorenzo
Version: Production → unspecified

The Purple SRE team owns this part of the infrastructure. Moving this bug over to them.

Component: General → Operations: Product Delivery
Product: Release Engineering → Cloud Services
QA Contact: jlorenzo

It’s even reproducible:

$ wget -O /dev/null https://ftp.mozilla.org/pub/firefox/releases/128.0.3/source/firefox-128.0.3.source.tar.xz
--2024-07-31 11:17:53--  https://ftp.mozilla.org/pub/firefox/releases/128.0.3/source/firefox-128.0.3.source.tar.xz
Resolving ftp.mozilla.org... 34.117.35.28
Connecting to ftp.mozilla.org|34.117.35.28|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 563698520 (538M) [application/octet-stream]
Saving to: ‘/dev/null’

/dev/null            14%[=>                  ]  80.21M  3.89MB/s    eta 1m 45s ^C

A little better today, but capped at 15.5 MB/s, and a network error:

@platsch:~$ wget -O /dev/null https://ftp.mozilla.org/pub/firefox/releases/128.0.3/source/firefox-128.0.3.source.tar.xz
--2024-08-14 22:02:26--  https://ftp.mozilla.org/pub/firefox/releases/128.0.3/source/firefox-128.0.3.source.tar.xz
Resolving ftp.mozilla.org... 34.117.35.28
Connecting to ftp.mozilla.org|34.117.35.28|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 563698520 (538M) [application/octet-stream]
Saving to: ‘/dev/null’

/dev/null            80%[===============>    ] 434.56M  15.0MB/s    in 30s     

2024-08-14 22:02:56 (14.7 MB/s) - Read error at byte 455664961/563698520 (The TLS connection was non-properly terminated.). Retrying.

--2024-08-14 22:02:57--  (try: 2)  https://ftp.mozilla.org/pub/firefox/releases/128.0.3/source/firefox-128.0.3.source.tar.xz
Connecting to ftp.mozilla.org|34.117.35.28|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 563698520 (538M), 108033559 (103M) remaining [application/octet-stream]
Saving to: ‘/dev/null’

/dev/null           100%[++++++++++++++++===>] 537.58M  16.1MB/s    in 7.3s    

2024-08-14 22:03:04 (14.1 MB/s) - ‘/dev/null’ saved [563698520/563698520]

The system has a 10 Gbit/s network device, and the Internet connection is at least 6 Gbit/s.

No problem with 129.0.1:

$ wget -O /dev/null https://ftp.mozilla.org/pub/firefox/releases/129.0.1/source/firefox-129.0.1.source.tar.xz
--2024-08-14 22:05:51--  https://ftp.mozilla.org/pub/firefox/releases/129.0.1/source/firefox-129.0.1.source.tar.xz
Resolving ftp.mozilla.org... 34.117.35.28
Connecting to ftp.mozilla.org|34.117.35.28|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 561709736 (536M) [application/octet-stream]
Saving to: ‘/dev/null’

/dev/null                                      100%[=================================================>] 535.69M  97.0MB/s    in 5.6s    

2024-08-14 22:05:56 (96.2 MB/s) - ‘/dev/null’ saved [561709736/561709736]

Same problem with 130.0:

--2024-09-09 08:28:33--  https://ftp.mozilla.org/pub/firefox/releases/130.0/source/firefox-130.0.source.tar.xz
Resolving ftp.mozilla.org... 34.117.35.28
Connecting to ftp.mozilla.org|34.117.35.28|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 568295588 (542M) [application/octet-stream]
Saving to: '/dev/shm/bee-pmenzel/firefox/files/firefox-130.0.source.tar.xz'
2024-09-09 08:30:40 (4.31 MB/s) - '/dev/shm/bee-pmenzel/firefox/files/firefox-130.0.source.tar.xz' saved [568295588/568295588]

Hmm, doing it again for 130.0, it was fast again. But 128.0.3 still has problems:

@hotelcalifornia:~$ wget -O /dev/null https://ftp.mozilla.org/pub/firefox/releases/128.0.3/source/firefox-128.0.3.source.tar.xz
--2024-09-09 08:32:12--  https://ftp.mozilla.org/pub/firefox/releases/128.0.3/source/firefox-128.0.3.source.tar.xz
Resolving ftp.mozilla.org... 34.117.35.28
Connecting to ftp.mozilla.org|34.117.35.28|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 563698520 (538M) [application/octet-stream]
Saving to: '/dev/null'

/dev/null            87%[================>   ] 469.83M  15.7MB/s    in 30s

2024-09-09 08:32:42 (15.8 MB/s) - Read error at byte 492657992/563698520 (The TLS connection was non-properly terminated.). Retrying.

--2024-09-09 08:32:43--  (try: 2)  https://ftp.mozilla.org/pub/firefox/releases/128.0.3/source/firefox-128.0.3.source.tar.xz
Connecting to ftp.mozilla.org|34.117.35.28|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 563698520 (538M), 71040528 (68M) remaining [application/octet-stream]
Saving to: '/dev/null'

/dev/null           100%[+++++++++++++++++==>] 537.58M  4.68MB/s    in 15s

2024-09-09 08:32:59 (4.58 MB/s) - '/dev/null' saved [563698520/563698520]

@hotelcalifornia:~$ wget -O /dev/null https://ftp.mozilla.org/pub/firefox/releases/130.0/source/firefox-130.0.source.tar.xz
--2024-09-09 08:33:57--  https://ftp.mozilla.org/pub/firefox/releases/130.0/source/firefox-130.0.source.tar.xz
Resolving ftp.mozilla.org... 34.117.35.28
Connecting to ftp.mozilla.org|34.117.35.28|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 568295588 (542M) [application/octet-stream]
Saving to: '/dev/null'

/dev/null                                   100%[===========================================================================================>] 541.97M   146MB/s    in 3.8s    

2024-09-09 08:34:01 (144 MB/s) - '/dev/null' saved [568295588/568295588]

@hotelcalifornia:~$ wget -O /dev/null https://ftp.mozilla.org/pub/firefox/releases/128.0.3/source/firefox-128.0.3.source.tar.xz
--2024-09-09 08:34:09--  https://ftp.mozilla.org/pub/firefox/releases/128.0.3/source/firefox-128.0.3.source.tar.xz
Resolving ftp.mozilla.org... 34.117.35.28
Connecting to ftp.mozilla.org|34.117.35.28|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 563698520 (538M) [application/octet-stream]
Saving to: '/dev/null'

/dev/null                                     5%[====>                                                                                       ]  30.05M  5.34MB/s    eta 1m 58s ^C

PS: I also got The TLS connection was non-properly terminated two times today already.

Same behavior from a 100 Mbit/s Vodafone Cable connection:

$ wget -O /dev/null https://ftp.mozilla.org/pub/firefox/releases/128.0.3/source/firefox-128.0.3.source.tar.xz
--2024-09-09 08:39:00--  https://ftp.mozilla.org/pub/firefox/releases/128.0.3/source/firefox-128.0.3.source.tar.xz
Auflösen des Hostnamens ftp.mozilla.org (ftp.mozilla.org)… 34.117.35.28
Verbindungsaufbau zu ftp.mozilla.org (ftp.mozilla.org)|34.117.35.28|:443 … verbunden.
HTTP-Anforderung gesendet, auf Antwort wird gewartet … 200 OK
Länge: 563698520 (538M) [application/octet-stream]
Wird in »/dev/null« gespeichert.

/dev/null            43%[=======>            ] 236,22M  4,75MB/s    ETA 63s    ^C
$ date; wget -O /dev/null https://ftp.mozilla.org/pub/firefox/releases/130.0/source/firefox-130.0.source.tar.xz
Mo 9. Sep 08:40:18 CEST 2024
--2024-09-09 08:40:18--  https://ftp.mozilla.org/pub/firefox/releases/130.0/source/firefox-130.0.source.tar.xz
Auflösen des Hostnamens ftp.mozilla.org (ftp.mozilla.org)… 34.117.35.28
Verbindungsaufbau zu ftp.mozilla.org (ftp.mozilla.org)|34.117.35.28|:443 … verbunden.
HTTP-Anforderung gesendet, auf Antwort wird gewartet … 200 OK
Länge: 568295588 (542M) [application/octet-stream]
Wird in »/dev/null« gespeichert.

/dev/null            55%[==========>         ] 298,69M  12,1MB/s    ETA 22s    ^C
You need to log in before you can comment on or make changes to this bug.