Closed Bug 393714 Opened 15 years ago Closed 15 years ago

ftp.m.o slow to sync

Categories

(mozilla.org Graveyard :: Server Operations, task)

task
Not set
major

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: nthomas, Assigned: aravind)

References

()

Details

Attachments

(1 obsolete file)

This comes via Jim & #firefox, who noticed that the tinderbox builds for Firefox trunk are stale on ftp.m.o. That box continues to produce builds happily but they're not making it from the staging machine to the public facing ftp site. (As I post this they are up to date, but were an hour old just a few minutes ago). See
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/FXNEWREF-WIN32--trunk/

The ftplag graph on nagios indicates a problem too, although I haven't see any other warnings.
Summary: ftp.m.o out of sync → ftp.m.o slow to sync
We have been experiencing high load on the server lately.  I already have a bug open on it.  I am guessing this is a direct result of that.
Assignee: server-ops → aravind
Depends on: 393462
I am currently running these rsyncs manually and hence the delay.  The high load on the server may have been caused by a combination of hardware and overlapping rsyncs.  Working on a way to speed this up.  Will post on the bug once I figure out a way.
Aravind, I added some excludes for the mozilla-all module (see surf:/etc/rsyncd-mozilla-all.exclude) but they don't seem to be working. eg this is still present
  http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/1.5.0.9-candidates/

Do you need to tweak the rsync on manna ?
This patch allows us to push ${app}/nightly/contrib directories to archive.m.o, which can save us the majority of 85 GB (out of a 580 GB module size). I've checked that all the target directories are already present, and done some testing with a simulacrum of the directory structure.

There is another 86 GB in calendar/{lightning,sunbird}/nightly, the majority of which can go over too. The find arguments exclude the 200{5,6,7} directories that those guys have already tidied into, so that's a manual push.
Attachment #278468 - Flags: review?
Attachment #278468 - Flags: review? → review?(justdave)
It was my impression that we already had a script running in there that outright deleted stuff in the contrib directories after a certain age...
Looks like /usr/local/sbin/ftp-trim-contrib.sh would do that for firefox and mozilla dirs, but it's disabled in crontab.
Added a new (well pilfered from cvs-sync) script to sync stage with manna.  This one uses a lock file, so stuff wont overrun each other.  This runs every 10 minutes and should take care of getting the nightlies synced ASAP.

Trimming the module sizes would still help :)
Bug#390479 is tracking work trimming rsync module.
The new scheme for rsyncing seems to be pretty effective, lag is up to 30 minutes and probably averages out to somwhere less than 15. That's a bit longer than in the past but should be ok. Time to close this as FIXED ?


(In reply to comment #8)
> Bug#390479 is tracking work trimming rsync module.

That's a different module actually.

The locked rsyncs have been keeping up pretty well for syncing stuff over.  Do we still need this bug or can I resolve it?
Closing this since the main issue is resolved and there is a different bug to trim ftp module size.
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
Attachment #278468 - Attachment is obsolete: true
Attachment #278468 - Flags: review?(justdave)
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.