pp-web0[1-2].phx.mozilla.com:Disk - All is WARNING: DISK WARNING

RESOLVED FIXED

Status

Infrastructure & Operations
WebOps: Other
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: dgarvey, Assigned: jakem)

Tracking

Details

(Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/389] )

(Reporter)

Description

3 years ago
I check /var/log and /home but couldn't find any easy pickins.

[dgarvey@pp-web02 ~]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      125G  106G   14G  90% /
/dev/sda1              99M   24M   71M  25% /boot
tmpfs                 3.0G     0  3.0G   0% /dev/shm
10.8.80.10:/symbols    12T  8.1T  4.0T  68% /mnt/socorro/symbols
10.8.80.10:/static_cluster/prod
                       50G  553M   50G   2% /mnt/static/prod
[dgarvey@pp-web02 ~]$

Updated

3 years ago
Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/389]
(Reporter)

Comment 1

3 years ago
And this one too.

[dgarvey@pp-web01 ~]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      125G  106G   14G  90% /
/dev/sda1              99M   30M   65M  32% /boot
tmpfs                 3.0G     0  3.0G   0% /dev/shm
10.8.80.10:/symbols    12T  8.1T  4.0T  68% /mnt/socorro/symbols
10.8.80.10:/static_cluster/prod
                       50G  554M   50G   2% /mnt/static/prod
[dgarvey@pp-web01 ~]$
Summary: pp-web02.phx.mozilla.com:Disk - All is WARNING: DISK WARNING → pp-web0[1-2].phx.mozilla.com:Disk - All is WARNING: DISK WARNING
100237988       /data/www/.git

This is supposed to be solved in cron:

[root@pp-web01 www]# ls -aFl /etc/cron.d/weekly-git-gc 
-rw-r--r-- 1 root root 57 Sep  4  2013 /etc/cron.d/weekly-git-gc
[root@pp-web01 www]# cat /etc/cron.d/weekly-git-gc 
0 0 * * 6 root cd /data/www; nice /usr/bin/git gc --quiet

but doesn't actually seem to have been run in the current logging cycle:

[root@pp-web01 www]# fgrep git /var/log/cron*
[root@pp-web01 www]# 


Running git gc manually on both machines. The cronjob needs fixing still, though.
Also running git gc on pp-web03, which was about to start alerting.
git gcs completed.

[root@pp-web01 www]# df /
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                     130805104  14132000 109921260  12% /

[root@pp-web02 .git]# df /
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                     130805104  14177940 109875320  12% /

[root@pp-web03 www]# df /
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                     130805104  17924056 106129204  15% /
(Assignee)

Comment 5

3 years ago
I'm speculating the reason this never runs is the lack of a trailing newline in /etc/cron.d/weekly-git-gc. I've added one, and we should check next week to see if it gets logged in /var/log/cron.

Comment 6

3 years ago
FYI, I ran the git gc again on all web nodes (pp-web01 - pp-web04) and the admin node for this cluster. This reduced the .git directory by about 95GB on them. The cron job is still not running but I see no obvious reason as to why. I just noticed this during the GHOST reboots.

Comment 7

3 years ago
It seems we were all working on the same thing this morning :)
(Assignee)

Comment 8

3 years ago
I tweaked this some more (into an actual script), and just now verified it did in fact run this past Saturday as it should.
Assignee: server-ops-webops → nmaul
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.