876150 - node40.seamicro.phx1.mozilla.com low on inodes on /

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Eric Ziegenhorn :ericz]

node40.seamicro.phx1 is alerting on inodes:

[node40.seamicro.phx1 ~]$ df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/vg_node40-lv_root
                      628320  591247   37073   95% /

Comment 1

[Eric Ziegenhorn :ericz]

Inventory shows no ownership info for this box, but it is serving websites and has these in /data/www:

[[e-mail]1 httpd]$ ls -l /data/www/
total 16
drwxr-xr-x 3 root root 4096 Jan 25 15:45 affiliates-dev.allizom.org
drwxr-xr-x 3 root root 4096 Feb 14  2012 detodosparatodos-dev.allizom.org
drwxr-xr-x 3 root root 4096 May 10 11:10 reps-dev.allizom.org
drwxr-xr-x 3 root root 4096 Aug  9  2011 webifyme-dev.allizom.org

Passing to webops for assistance in what can be cleaned up (or if we just need to redo this server with more inodes).  There are 31k logs in /var/log/httpd which might be some low-hanging fruit (though probably not enough on their own).

Comment 2

[Chris Turra [:cturra]]

i just kicked off a git gc (garbage collection) job on this node, which should help us out here. will keep this bug posted once it's done -- could take some time.

Comment 3

[Eric Ziegenhorn :ericz]

Awesome.  Do you do that in /data/www?  Also, can you update https://inventory.mozilla.org/en-US/systems/show/3043/ to whatever it should be, perhaps "infra : webops"?

Comment 4

[Chris Turra [:cturra]]

(In reply to Eric Ziegenhorn :ericz from comment #3)
> Awesome.  Do you do that in /data/www?  

looking through dns, it appears this node is a dev celery node for the engagement cluster. *this is also noted in inventory*

$ host engagement-celery1.dev.seamicro.phx1.mozilla.com
engagement-celery1.dev.seamicro.phx1.mozilla.com is an alias for node40.seamicro.phx1.mozilla.com.
node40.seamicro.phx1.mozilla.com has address 10.8.32.46


> Also, can you update https://inventory.mozilla.org/en-US/systems/show/3043/ to 
> whatever it should be, perhaps "infra : webops"?

updated.

Comment 5

[Chris Turra [:cturra]]

git garbage collection is complete. the disk usage is looking much better.

[[e-mail]1 ~]# cd /data/www/

[[e-mail]1 www]# git gc
Counting objects: 868445, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (525589/525589), done.
Writing objects: 100% (868445/868445), done.
Total 868445 (delta 707773), reused 422795 (delta 337545)
Removing duplicate objects: 100% (256/256), done.
Checking connectivity: 868553, done.

[[e-mail]1 www]# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/vg_node40-lv_root
                      628320  153838  474482   25% /


i have also gone ahead and applied a cron to this node so that `git gc` is run weekly to keep this from happening again.

[[e-mail]1 www]# puppetctl run --test
Info: Retrieving plugin
...
Info: Caching catalog for node40.seamicro.phx1.mozilla.com
Info: Applying configuration version '66655'
Notice: /Stage[main]/Webapp::Engagement::Dev/File[/etc/cron.d/weekly-git-gc]/ensure: defined content as '{md5}3a4294825179bba7184826fde1cb19e6'
Notice: Finished catalog run in 101.78 seconds

[[e-mail]1 www]# more /etc/cron.d/weekly-git-gc
0 0 * * 6 root cd /data/www; nice /usr/bin/git gc --quiet