1128432 - pp-web0[1-2].phx.mozilla.com:Disk - All is WARNING: DISK WARNING

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[david garvey:dgarvey]

I check /var/log and /home but couldn't find any easy pickins.

[dgarvey@pp-web02 ~]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      125G  106G   14G  90% /
/dev/sda1              99M   24M   71M  25% /boot
tmpfs                 3.0G     0  3.0G   0% /dev/shm
10.8.80.10:/symbols    12T  8.1T  4.0T  68% /mnt/socorro/symbols
10.8.80.10:/static_cluster/prod
                       50G  553M   50G   2% /mnt/static/prod
[dgarvey@pp-web02 ~]$

Comment 1

[david garvey:dgarvey]

And this one too.

[dgarvey@pp-web01 ~]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      125G  106G   14G  90% /
/dev/sda1              99M   30M   65M  32% /boot
tmpfs                 3.0G     0  3.0G   0% /dev/shm
10.8.80.10:/symbols    12T  8.1T  4.0T  68% /mnt/socorro/symbols
10.8.80.10:/static_cluster/prod
                       50G  554M   50G   2% /mnt/static/prod
[dgarvey@pp-web01 ~]$

Comment 2

[Peter Radcliffe [:pir]]

100237988       /data/www/.git

This is supposed to be solved in cron:

[root@pp-web01 www]# ls -aFl /etc/cron.d/weekly-git-gc 
-rw-r--r-- 1 root root 57 Sep  4  2013 /etc/cron.d/weekly-git-gc
[root@pp-web01 www]# cat /etc/cron.d/weekly-git-gc 
0 0 * * 6 root cd /data/www; nice /usr/bin/git gc --quiet

but doesn't actually seem to have been run in the current logging cycle:

[root@pp-web01 www]# fgrep git /var/log/cron*
[root@pp-web01 www]# 


Running git gc manually on both machines. The cronjob needs fixing still, though.

Comment 3

[Peter Radcliffe [:pir]]

Also running git gc on pp-web03, which was about to start alerting.

Comment 4

[Peter Radcliffe [:pir]]

git gcs completed.

[root@pp-web01 www]# df /
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                     130805104  14132000 109921260  12% /

[root@pp-web02 .git]# df /
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                     130805104  14177940 109875320  12% /

[root@pp-web03 www]# df /
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                     130805104  17924056 106129204  15% /

Comment 5

[Jake Maul [:jakem]]

I'm speculating the reason this never runs is the lack of a trailing newline in /etc/cron.d/weekly-git-gc. I've added one, and we should check next week to see if it gets logged in /var/log/cron.

Comment 6

[Jason Crowe [:jd]]

FYI, I ran the git gc again on all web nodes (pp-web01 - pp-web04) and the admin node for this cluster. This reduced the .git directory by about 95GB on them. The cron job is still not running but I see no obvious reason as to why. I just noticed this during the GHOST reboots.

Comment 7

[Jason Crowe [:jd]]

It seems we were all working on the same thing this morning :)

Comment 8

[Jake Maul [:jakem]]

I tweaked this some more (into an actual script), and just now verified it did in fact run this past Saturday as it should.