reduce frequency of du's done for rsync module MOTDs


Status Graveyard
Server Operations
6 years ago
3 years ago


(Reporter: bhearsum, Assigned: nthomas)





6 years ago
Given how strained the netapp is, I think it would be good to reduce the number of times we run big du's for the rsync module MOTDs. Right now we run the following every night:
MOZILLAALL=`du -sh /mnt/netapp/stage/ | sed 's/\s.*$//'`
# the master is in /data/ftp-releases/pub/ but there's an exclude file on the module
# which eliminates about 15GB of stuff, so use an rsync-generated copy from the netapp instead
MOZILLARELEASE=`du -sh /mnt/netapp/stage/ | sed 's/\s.*$//'`
MOZILLACURRENT=`du -sh /mnt/netapp/stage/ | sed 's/\s.*$//'`

Right now, it's been running the first one for 6 hours and still hasn't completed.

If we could run these once a week, maybe on saturday or sunday morning, I think that would help a lot.

Comment 1

6 years ago
I wonder if we're better off turning this down: (surf:/etc/cron.d/ftp-staging-rw-server)
 # update releases mirror on the netapp
 */15 * * * * root /root/bin/sync-stage1-releases
 */15 * * * * root /root/bin/sync-stage1-current

It has locking but if I/O is lagging it's not useful enough to keep doing on that frequency. The downside of only doing the du once a week is that I'll forget to trim the rsync module, via the nagios check, but I'm all ears if you've got a better way to monitor that.

Comment 2

6 years ago
I've updated surf:/etc/cron.daily/ to not calculate MOZILLAALL, since that's a number which is very expensive to calculate and we don't use it for anything useful.

I've also updated surf:/etc/cron.d/ftp-staging-rw-server to disable the every-15-minute cron, and make those calls from instead.
So are we good here? Nick, can you take the bug and eventually close it?
Or let us know how can we help :)

Comment 4

6 years ago
I've done a bit more so that we can get back to a monitored state. Namely put the mozilla-prereleases in the motd:

            R E L E A S E S - R S Y N C . M O Z I L L A . O R G

                   Current module sizes (updated 2012-02-14)

                   mozilla-prereleases: 227GB
                      mozilla-releases: 116GB
                       mozilla-current: 26GB



surf:/etc/cron.daily/ is the master script for that. There's ionice sprinkled liberally around, and it only runs once a day (4am) to keep the load down. Bug 723815 to use that in nagios.

Lets stick a stake in this one.
Assignee: server-ops → nrthomas
Last Resolved: 6 years ago
Resolution: --- → FIXED
Product: → Graveyard
You need to log in before you can comment on or make changes to this bug.