652335 - cm-metricsapp02 is out of disk space

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Dietrich Ayala (:dietrich)]

Now that's no way to celebrate 100 million downloads!

Comment 1

[Jeff Balogh (:jbalogh)]

cm-metricsapp02 is out of disk space.

[jbalogh@cm-metricsapp02 home]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      100G   95G  154M 100% /
/dev/cciss/c0d0p1      99M   26M   68M  28% /boot
tmpfs                  16G     0   16G   0% /dev/shm
10.253.0.121:/data    8.1T  7.8T  299G  97% /mnt/stats_im-log02
10.253.0.65:/data     774G  421G  315G  58% /mnt/stats_im-log03
10.253.0.121:/data-2/stats/
                      7.2T  3.3T  3.9T  46% /data-2/stats
10.253.0.11:/vol/stats_nfs
                      600G  496G  105G  83% /mnt/stats
10.253.0.11:/vol/mpt_app_cluster/glow.mozilla.org
                      288G  251G   38G  87% /mnt/netapp/glow.mozilla.org

Comment 2

[Jeff Balogh (:jbalogh)]

I got glow going again by deleting my 250M log file, but it's going to crash again soon.

Comment 3

[Daniel Einspanjer [:dre] [:deinspanjer]]

The rotate logs cron was disabled during some troubleshooting a while back and wasn't restarted.
Also, there was a rsyslog filter statement missing which caused all the dmo log data to be copied to /var/log/messages.