Closed Bug 1238601 Opened 10 years ago Closed 10 years ago

netops1.private.scl3.mozilla.com:NetOps Logstash - health check is CRITICAL - FILE_AGE CRITICAL: /var/log/logstash/all.log is 45627 seconds old and 65177263

Categories

(Infrastructure & Operations Graveyard :: NetOps: Other, task)

Product:
Infrastructure & Operations Graveyard
Component:
NetOps: Other
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: mlankford, Unassigned)

Details

8:12 AM <@nagios-scl3> mlankford: netops1.private.scl3.mozilla.com:NetOps Logstash - health check is CRITICAL - FILE_AGE CRITICAL: /var/log/logstash/all.log is 45627 seconds old and 65177263 bytes Last Checked: 2016-01-11 07:57:30 PST
root@netops1.private.scl3 logstash]# ls -ld /var/log/logstash/all.log-20160111 -rw-r--r-- 1 root root 65177263 Jan 11 03:17 /var/log/logstash/all.log-20160111 [root@netops1.private.scl3 logstash]# ps ax | grep logstash.jar 10872 ? Sl 757:39 /usr/bin/java -Dcom.sun.management.jmxremote.port=3333 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Xmx4096M -Xms1536M -jar logstash.jar agent -f /opt/logstash/etc//logstash.conf --log /var/log/logstash/logstash.log 24428 pts/0 S+ 0:00 grep logstash.jar [root@netops1.private.scl3 logstash]# kill 10872 [root@netops1.private.scl3 logstash]# ps ax | grep logstash.jar 24456 pts/0 S+ 0:00 grep logstash.jar [root@netops1.private.scl3 logstash]# ls -al /var/log/logstash/all.log lrwxrwxrwx 1 root root 34 Jan 11 00:00 /var/log/logstash/all.log -> /var/log/logstash/all.log-20160111 [root@netops1.private.scl3 logstash]# ls -ltr total 1458812 -rw-r--r-- 1 root root 0 Feb 10 2013 input.log -rw-r--r-- 1 root root 549491 Jul 24 2014 logstash.log.bz2 -rw-r--r-- 1 root root 38894586 Dec 29 23:59 all.log-20151229.gz -rw-r--r-- 1 root root 39026185 Dec 30 23:59 all.log-20151230.gz -rw-r--r-- 1 root root 39315235 Dec 31 23:59 all.log-20151231.gz -rw-r--r-- 1 root root 39177789 Jan 1 23:59 all.log-20160101.gz -rw-r--r-- 1 root root 39028875 Jan 2 23:59 all.log-20160102.gz -rw-r--r-- 1 root root 39099087 Jan 3 23:59 all.log-20160103.gz -rw-r--r-- 1 root root 39351594 Jan 4 23:59 all.log-20160104.gz -rw-r--r-- 1 root root 38973470 Jan 5 23:59 all.log-20160105.gz -rw-r--r-- 1 root root 38904251 Jan 6 23:59 all.log-20160106.gz -rw-r--r-- 1 root root 5688764 Jan 7 03:32 all.log-20160107.gz -rw-r--r-- 1 root root 22838606 Jan 7 03:32 logstash.log-20160107 -rw-r--r-- 1 root root 181067556 Jan 7 23:59 all.log-20160107 -rw-r--r-- 1 root root 5899051 Jan 8 03:29 all.log-20160108.gz -rw-r--r-- 1 root root 286 Jan 8 03:29 logstash.log-20160108.gz -rw-r--r-- 1 root root 306158286 Jan 8 23:59 all.log-20160108 -rw-r--r-- 1 root root 5782110 Jan 9 03:33 all.log-20160109.gz -rw-r--r-- 1 root root 286 Jan 9 03:33 logstash.log-20160109.gz -rw-r--r-- 1 root root 151918607 Jan 9 23:59 all.log-20160109 -rw-r--r-- 1 root root 5838421 Jan 10 03:36 all.log-20160110.gz -rw-r--r-- 1 root root 275 Jan 10 03:36 logstash.log-20160110.gz -rw-r--r-- 1 root root 391055757 Jan 10 23:59 all.log-20160110 lrwxrwxrwx 1 root root 34 Jan 11 00:00 all.log -> /var/log/logstash/all.log-20160111 -rw-r--r-- 1 root root 409 Jan 11 03:17 logstash.log-20160111.gz -rw-r--r-- 1 root root 891 Jan 11 16:16 logstash.log -rw-r--r-- 1 root root 65179179 Jan 11 16:16 all.log-20160111 [root@netops1.private.scl3 logstash]#
This apparently got fixed at some point, the problem is no longer present.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → WORKSFORME
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.