Automated alert report from nagios1.private.scl3.mozilla.com: Hostname: mozdefes1.private.scl3.mozilla.com Service: Disk - All State: WARNING Output: DISK WARNING - free space: /data 459420 MB (10% inode=99%): Runbook: http://m.allizom.org/Disk+-+All
CC'ing :jeff - please prune some indexes. Thanks!
Automated alert acknowledgement: (ashish)Bug 1026839
Automated alert recovery: Hostname: mozdefes1.private.scl3.mozilla.com Service: Disk - All State: OK Output: DISK OK
Some of this is due to the timing of backups vs pruning. I'll see about re-arranging things.
Any update on this? We are still getting alerts (flapping; alerts, recovers, alerts etc).
Updated the timings of backup/purge. Some of this was due to sudden surge in errant data resulting in logging loads of requests. Fix is in progress and we are working new backup/purge/maintenance scripts, and estimates for additional disk.