850407 - genericadm.private.phx1 disk is running out of space

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P3)

Description

[Eric Ziegenhorn :ericz]

Similar to bug 725826, genericadm.private.phx1 is running out of disk space on /.  Most of it seems to be in /data.

< nagios-phx1> | Tue 14:09:44 PDT [136] genericadm.private.phx1.mozilla.com:Disk - All is WARNING: DISK WARNING - free space: / 5012 MB

Comment 1

[Ashish Vijayaram [:ashish]]

There is a /data/bin/git-cleanup.sh script that runs every week. Probably a good idea to run that twice maybe?

FWIW I've kicked off a run by hand.

Comment 2

[Ashish Vijayaram [:ashish]]

(In reply to Ashish Vijayaram [:ashish] from comment #1)
> There is a /data/bin/git-cleanup.sh script that runs every week. Probably a
> good idea to run that twice maybe?
> 
> FWIW I've kicked off a run by hand.

and then OTOH looks like / is actually running out of disk:

After:
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       57G   50G  4.6G  92% /

Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/VolGroup00-LogVol00
                     3792896 2555830 1237066   68% /

Comment 3

[Brandon Burton [:solarce]]

We're going to work with Storage to grow the disk

Comment 4

[Rick Bryce [:rbryce]]

Getting full again.

genericadm.private.phx1.mozilla.com:Disk - All is CRITICAL: DISK CRITICAL - free space: / 1852 MB (3% inode=30%): (http://m.allizom.org/Disk+-+All) 18:33:48

Comment 5

[Jake Maul [:jakem]]

<3's all around.