119665 - rsync on stage.mozilla.org is unusably slow

Status: RESOLVED FIXED

(mozilla.org :: FTP: Staging, task)

Description

[Tuukka Tolvanen (sp3000)]

directories in http://ftp.mozilla.org/pub/mozilla/nightly/ since
2002-01-11-13-trunk (11-Jan-2002 13:49) contain no files, just some empty dirs
here and there

Comment 1

[Dawn Endico]

Ray, can you look at this?

rsync is being phenomenally slow. it recieves the file
list, creates the directories it needs and then gets 
hung up on very slow file transfers.

when i started looking at this there were more than 135
rsync processes running. I killed them all and disabled
rsync in the ftp-rsync user's crontab file. I'm trying
to run one rsync by hand now but its taking forever.

The file it tries to download first is 
/pub/mozilla/nightly/2002-01-11-16-trunk/mozilla-os2.zip
This file is 15mb and after an hour the transfer is less
than half complete.

I don't see why the transfer should be so slow. Pings from
komodo to stage.mozilla.org seem pretty speedy. I scp'd
files from my home machine to both stage and komodo and got
normal speeds.

I thought maybe stage's rsync daemon process was horked but
its started by inetd and restarts every time.

Something like this happened a few days ago (dozens of
rsync processes) but the problem went away on its own after
a little while.

----

Sonja and wtc,

I noticed that there were an extreme number of files under
mozilla/security/nss and thought that might be contributing
to rsync's slowness.  The tinderbox directory had >1500 
files in it. the daily qa directory was taking up a lot of
space and had 3400 files in sub-directories under it. (I'm
not sure whether they were any single directories with a
lot of files though)

I moved these two directories to /home/sonmi on stage
and sonmi's script quickly repopulated them under /home/ftp,
but time with what seems to be a more reasonable amount of
stuff. I think you were deleting old stuff on your local
disk but stuff wasn't getting deleted on stage. I think
you have to tell scp explicitly to delete files when you
scp -r a directory.

Comment 2

[Dawn Endico]

this is keeping people from testing new builds. marking as 
critical for now but this will definately be a blocker by
monday.

Comment 3

[Risto Kotalampi]

I can see 20-40% packet drop with bigger packets from komodo to stage. Please 
run : /usr/sbin/ping -s stage 1500, and see it yourself.

Comment 4

[Risto Kotalampi]

And I don't see similar packet loss elsewhere, for example my home ADSL. (Just 
to see if it's local end giving drops).

Comment 5

[Risto Kotalampi]

----stage.mozilla.org PING Statistics----
6279 packets transmitted, 5034 packets received, 19% packet loss
round-trip (ms)  min/avg/max = 7/8/224

Comment 6

[Risto Kotalampi]

Moving to staging subcomponent.

Comment 7

[Dawn Endico]

i did a traceroute and did 'ping -s 1500' on some of the sites between
my home machine and stage and only got lost packets when pinging stage itself.

Comment 8

[Jeremy M. Dolan]

No comments here since Saturday... any progress? If rsync is still busted, can
we scp a few builds to ftp for today?

Comment 9

[Dawn Endico]

yes we can probably copy some builds to the ftp site by hand, but
not till the builds are created and people wake up and get in to work.

No word from the VA network admin yet and there probably won't be till
he wakes up and gets to work.

Comment 10

[Daniel (Leaf) Nunes]

dawn, if you give me the 411 (call my cell?) i'll push bits by hand today; i
don't know what user i'm supposed to use to scp.

Comment 11

[Sonja Mirtitsch]

Should I stop my scripts? They have not changed in 8 week or so though, neither
has the amount of files/directories in the staging area.

Comment 12

[Dawn Endico]

Yes, please! the morning builds should be ready so go ahead any time.
upload to /ftp-root/tmp. chown to everything to ftp-stage, chgrp to
ftp-stage and then mv in to place.

Comment 13

[Dawn Endico]

It turns out VA Linux had a bandwidth emergency over the weekend
and had us and the rest of their community machines (including
cvs-mirror) bandwidth  limited.  Things should be back to normal now.

'ping -s 1500' to both stage.mozilla.org and cvs-mirror.mozilla.org
are back to normal now. I was able to rsync the staging server in
a normal amount of time and I turned the rsync cron job back on.
Marking fixed.