reduce frequency of du's done for rsync module MOTDs

RESOLVED FIXED

Status

mozilla.org Graveyard
Server Operations
RESOLVED FIXED
6 years ago
3 years ago

People

(Reporter: bhearsum, Assigned: nthomas)

Tracking

Details

(Reporter)

Description

6 years ago
Given how strained the netapp is, I think it would be good to reduce the number of times we run big du's for the rsync module MOTDs. Right now we run the following every night:
MOZILLAALL=`du -sh /mnt/netapp/stage/archive.mozilla.org/pub/ | sed 's/\s.*$//'`
# the master is in /data/ftp-releases/pub/ but there's an exclude file on the module
# which eliminates about 15GB of stuff, so use an rsync-generated copy from the netapp instead
MOZILLARELEASE=`du -sh /mnt/netapp/stage/releases.mozilla.org/pub/ | sed 's/\s.*$//'`
MOZILLACURRENT=`du -sh /mnt/netapp/stage/currelease.mozilla.org/pub/ | sed 's/\s.*$//'`

Right now, it's been running the first one for 6 hours and still hasn't completed.

If we could run these once a week, maybe on saturday or sunday morning, I think that would help a lot.
(Assignee)

Comment 1

6 years ago
I wonder if we're better off turning this down: (surf:/etc/cron.d/ftp-staging-rw-server)
 # update releases mirror on the netapp
 */15 * * * * root /root/bin/sync-stage1-releases
 */15 * * * * root /root/bin/sync-stage1-current

It has locking but if I/O is lagging it's not useful enough to keep doing on that frequency. The downside of only doing the du once a week is that I'll forget to trim the rsync module, via the nagios check, but I'm all ears if you've got a better way to monitor that.
(Assignee)

Comment 2

6 years ago
I've updated surf:/etc/cron.daily/motd-gen.sh to not calculate MOZILLAALL, since that's a number which is very expensive to calculate and we don't use it for anything useful.

I've also updated surf:/etc/cron.d/ftp-staging-rw-server to disable the every-15-minute cron, and make those calls from motd-gen.sh instead.
So are we good here? Nick, can you take the bug and eventually close it?
Or let us know how can we help :)
Thanks.
(Assignee)

Comment 4

6 years ago
I've done a bit more so that we can get back to a monitored state. Namely put the mozilla-prereleases in the motd:

---------------------------------------------------------------------------
            R E L E A S E S - R S Y N C . M O Z I L L A . O R G
---------------------------------------------------------------------------

                   Current module sizes (updated 2012-02-14)

                   mozilla-prereleases: 227GB
                      mozilla-releases: 116GB
                       mozilla-current: 26GB

         Questions/comments/complaints: sysadmins@mozilla.org

---------------------------------------------------------------------------

surf:/etc/cron.daily/motd-gen.sh is the master script for that. There's ionice sprinkled liberally around, and it only runs once a day (4am) to keep the load down. Bug 723815 to use that in nagios.

Lets stick a stake in this one.
Assignee: server-ops → nrthomas
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.