725711 - reduce frequency of du's done for rsync module MOTDs

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[[e-mail] (:bhearsum)]

Given how strained the netapp is, I think it would be good to reduce the number of times we run big du's for the rsync module MOTDs. Right now we run the following every night:
MOZILLAALL=`du -sh /mnt/netapp/stage/archive.mozilla.org/pub/ | sed 's/\s.*$//'`
# the master is in /data/ftp-releases/pub/ but there's an exclude file on the module
# which eliminates about 15GB of stuff, so use an rsync-generated copy from the netapp instead
MOZILLARELEASE=`du -sh /mnt/netapp/stage/releases.mozilla.org/pub/ | sed 's/\s.*$//'`
MOZILLACURRENT=`du -sh /mnt/netapp/stage/currelease.mozilla.org/pub/ | sed 's/\s.*$//'`

Right now, it's been running the first one for 6 hours and still hasn't completed.

If we could run these once a week, maybe on saturday or sunday morning, I think that would help a lot.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

I wonder if we're better off turning this down: (surf:/etc/cron.d/ftp-staging-rw-server)
 # update releases mirror on the netapp
 */15 * * * * root /root/bin/sync-stage1-releases
 */15 * * * * root /root/bin/sync-stage1-current

It has locking but if I/O is lagging it's not useful enough to keep doing on that frequency. The downside of only doing the du once a week is that I'll forget to trim the rsync module, via the nagios check, but I'm all ears if you've got a better way to monitor that.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

I've updated surf:/etc/cron.daily/motd-gen.sh to not calculate MOZILLAALL, since that's a number which is very expensive to calculate and we don't use it for anything useful.

I've also updated surf:/etc/cron.d/ftp-staging-rw-server to disable the every-15-minute cron, and make those calls from motd-gen.sh instead.

Comment 3

[Dumitru Gherman [:dumitru]]

So are we good here? Nick, can you take the bug and eventually close it?
Or let us know how can we help :)
Thanks.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

I've done a bit more so that we can get back to a monitored state. Namely put the mozilla-prereleases in the motd:

---------------------------------------------------------------------------
            R E L E A S E S - R S Y N C . M O Z I L L A . O R G
---------------------------------------------------------------------------

                   Current module sizes (updated 2012-02-14)

                   mozilla-prereleases: 227GB
                      mozilla-releases: 116GB
                       mozilla-current: 26GB

         Questions/comments/complaints: [e-mail]

---------------------------------------------------------------------------

surf:/etc/cron.daily/motd-gen.sh is the master script for that. There's ionice sprinkled liberally around, and it only runs once a day (4am) to keep the load down. Bug 723815 to use that in nagios.

Lets stick a stake in this one.