393714 - ftp.m.o slow to sync

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

This comes via Jim & #firefox, who noticed that the tinderbox builds for Firefox trunk are stale on ftp.m.o. That box continues to produce builds happily but they're not making it from the staging machine to the public facing ftp site. (As I post this they are up to date, but were an hour old just a few minutes ago). See
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/FXNEWREF-WIN32--trunk/

The ftplag graph on nagios indicates a problem too, although I haven't see any other warnings.

Comment 1

[Aravind Gottipati [:aravind]]

We have been experiencing high load on the server lately.  I already have a bug open on it.  I am guessing this is a direct result of that.

Comment 2

[Aravind Gottipati [:aravind]]

I am currently running these rsyncs manually and hence the delay.  The high load on the server may have been caused by a combination of hardware and overlapping rsyncs.  Working on a way to speed this up.  Will post on the bug once I figure out a way.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Aravind, I added some excludes for the mozilla-all module (see surf:/etc/rsyncd-mozilla-all.exclude) but they don't seem to be working. eg this is still present
  http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/1.5.0.9-candidates/

Do you need to tweak the rsync on manna ?

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Push more nightlies builds to archive

This patch allows us to push ${app}/nightly/contrib directories to archive.m.o, which can save us the majority of 85 GB (out of a 580 GB module size). I've checked that all the target directories are already present, and done some testing with a simulacrum of the directory structure.

There is another 86 GB in calendar/{lightning,sunbird}/nightly, the majority of which can go over too. The find arguments exclude the 200{5,6,7} directories that those guys have already tidied into, so that's a manual push.

Comment 5

[Dave Miller [:justdave]]

It was my impression that we already had a script running in there that outright deleted stuff in the contrib directories after a certain age...

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Looks like /usr/local/sbin/ftp-trim-contrib.sh would do that for firefox and mozilla dirs, but it's disabled in crontab.

Comment 7

[Aravind Gottipati [:aravind]]

Added a new (well pilfered from cvs-sync) script to sync stage with manna.  This one uses a lock file, so stuff wont overrun each other.  This runs every 10 minutes and should take care of getting the nightlies synced ASAP.

Trimming the module sizes would still help :)

Comment 8

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Bug#390479 is tracking work trimming rsync module.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

The new scheme for rsyncing seems to be pretty effective, lag is up to 30 minutes and probably averages out to somwhere less than 15. That's a bit longer than in the past but should be ok. Time to close this as FIXED ?


(In reply to comment #8)
> Bug#390479 is tracking work trimming rsync module.

That's a different module actually.

Comment 10

[Aravind Gottipati [:aravind]]

The locked rsyncs have been keeping up pretty well for syncing stuff over.  Do we still need this bug or can I resolve it?

Comment 11

[Aravind Gottipati [:aravind]]

Closing this since the main issue is resolved and there is a different bug to trim ftp module size.