Closed Bug 1022057 Opened 10 years ago Closed 10 years ago

Disk - All on hgweb5.dmz.scl3.mozilla.com is WARNING: DISK WARNING - free space: / 28502 MB (10% inode=56%):

Categories

(mozilla.org Graveyard :: Server Operations: MOC, task)

Other
Other
task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: nagiosapi, Unassigned)

References

()

Details

(Whiteboard: [id=nagios1.private.scl3.mozilla.com:367088])

Automated alert report from nagios1.private.scl3.mozilla.com:

Hostname: hgweb5.dmz.scl3.mozilla.com
Service:  Disk - All
State:    WARNING
Output:   DISK WARNING - free space: / 28502 MB (10% inode=56%):

Runbook:  http://m.allizom.org/Disk+-+All
this is odd, i see people tailing the logs and it isn't getting rotated.

[dgarvey@hgweb5.dmz.scl3 ~]$ sudo -i lsof | grep deleted
ipwatchd   1854      root  txt       REG                8,3      27880   27093960 /usr/sbin/ipwatchd (deleted)
hald       5871 haldaemon  txt       REG                8,3     351136   27061082 /usr/sbin/hald (deleted)
hald-runn  5872      root  txt       REG                8,3      16792   27061081 /usr/libexec/hald-runner (deleted)
hald-addo  5902      root  txt       REG                8,3      24744   27061062 /usr/libexec/hald-addon-input.#prelink#.xMTvdg (deleted)
hald-addo  5917 haldaemon  txt       REG                8,3      15296   27061058 /usr/libexec/hald-addon-acpi.#prelink#.YTutS8 (deleted)
abrt-dump  6071      root  txt       REG                8,3      15352   27059807 /usr/bin/abrt-dump-oops (deleted)
rhnsd      6100      root  txt       REG                8,3      14384   27060852 /usr/sbin/rhnsd (deleted)
mingetty   6113      root  txt       REG                8,3      15256   45088070 /sbin/mingetty (deleted)
mingetty   6115      root  txt       REG                8,3      15256   45088070 /sbin/mingetty (deleted)
mingetty   6117      root  txt       REG                8,3      15256   45088070 /sbin/mingetty (deleted)
mingetty   6119      root  txt       REG                8,3      15256   45088070 /sbin/mingetty (deleted)
mingetty   6121      root  txt       REG                8,3      15256   45088070 /sbin/mingetty (deleted)
mingetty   6123      root  txt       REG                8,3      15256   45088070 /sbin/mingetty (deleted)
lldpd      7059      root  txt       REG                8,3     152122   27062694 /usr/sbin/lldpd (deleted)
lldpd      7062    _lldpd  txt       REG                8,3     152122   27062694 /usr/sbin/lldpd (deleted)
httpd      8623        hg   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd      8641        hg   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd      8669        hg   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd      8670        hg   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd      8687        hg   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd      8785        hg   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
dnsmasq    8796    nobody  txt       REG                8,3     258501   27066578 /usr/sbin/dnsmasq (deleted)
httpd      8805        hg   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd      8810        hg   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd      8817        hg   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd      8822        hg   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd      8827        hg   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd      8832        hg   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd      8841        hg   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd      8842        hg   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd      8852        hg   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd      8900        hg   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd      8910        hg   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd      8915        hg   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd      8921        hg   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd      8964        hg   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd      8975        hg   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd      8983        hg   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd      8989        hg   14wW     REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd      9004        hg   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd     10298      root   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
tail      11195      root    3r      REG                8,3 2915459359     724877 /var/log/httpd/hg.mozilla.org/access_log-20140427 (deleted)
nrpe      21655    nagios  txt       REG                8,3      48648   27115841 /usr/sbin/nrpe (deleted)
tail      29730      root    3r      REG                8,3    9129878     724085 /var/log/httpd/hg.mozilla.org/error_log-20140523 (deleted)
httpd     29854    apache   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd     29855    apache   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd     29856    apache   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd     29857    apache   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd     29859    apache   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd     29861    apache   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd     29862    apache   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd     29863    apache   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd     29939    apache   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd     29940    apache   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
tail      30462      root    3r      REG                8,3    9129878     724085 /var/log/httpd/hg.mozilla.org/error_log-20140523 (deleted)
tail      30478      root    3r      REG                8,3    9129878     724085 /var/log/httpd/hg.mozilla.org/error_log-20140523 (deleted)
tail      30564      root    3r      REG                8,3    9129878     724085 /var/log/httpd/hg.mozilla.org/error_log-20140523 (deleted)
httpd     32017    apache   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd     32289    apache   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
httpd     32489    apache   14w      REG                8,3          0     861989 /var/run/wsgi.10298.65.1.lock (deleted)
[dgarvey@hgweb5.dmz.scl3 ~]$
Automated alert acknowledgement: (dgarvey)I would like to restart apache
Status: NEW → ASSIGNED
Automated alert recovery:

Hostname: hgweb5.dmz.scl3.mozilla.com
Service:  Disk - All
State:    OK
Output:   DISK OK
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.