1081465 - Check received log volume on metrics-logger1.private.scl3.mozilla.com is CRITICAL: CRITICAL: Daily log size from /data/stats/logs/zlb11.ops.phx1.mozilla.com is low at 0MB compared to running average 149MB

Status: RESOLVED FIXED

(Infrastructure & Operations :: MOC: Problems, task)

Description

[MOC Nagios API]

Automated alert report from nagios1.private.scl3.mozilla.com:

Hostname: metrics-logger1.private.scl3.mozilla.com
Service:  Check received log volume
State:    CRITICAL
Output:   CRITICAL: Daily log size from /data/stats/logs/zlb11.ops.phx1.mozilla.com is low at 0MB compared to running average 149MB

Runbook:  http://m.allizom.org/Check+received+log+volume

Comment 1

[MOC Nagios API]

Automated alert acknowledgement: (dgarvey)bug1081465

Comment 2

[david garvey:dgarvey]

It look like the zeus LB is shutdown on this box and is not creating logs.

[[e-mail]1 ~]# ls -lrt /var/log/zeus
total 16704
-rw-r--r-- 1 root root       0 Jun 21 00:00 symbolpush.ssh_2014-06-21
-rw-r--r-- 1 root root       0 Jun 21 00:00 ldapproxy_client_cert_2014-06-21
-rw-r--r-- 1 root root       0 Jun 21 00:00 ldapproxy_2014-06-21
-rw-r--r-- 1 root root       0 Jun 21 00:00 db-internal-pg-rw.2014-06-21.log
-rw-r--r-- 1 root root 2243636 Jun 21 00:00 unified-http-to-https-redirect.2014-06-20.log
-rw-r--r-- 1 root root 2129466 Jun 22 00:00 unified-http-to-https-redirect.2014-06-21.log
-rw-r--r-- 1 root root       0 Jun 22 00:00 symbolpush.ssh_2014-06-22
-rw-r--r-- 1 root root       0 Jun 22 00:00 ldapproxy_client_cert_2014-06-22
-rw-r--r-- 1 root root       0 Jun 22 00:00 ldapproxy_2014-06-22
-rw-r--r-- 1 root root       0 Jun 22 00:00 db-internal-pg-rw.2014-06-22.log
-rw-r--r-- 1 root root 2292012 Jun 23 00:00 unified-http-to-https-redirect.2014-06-22.log
-rw-r--r-- 1 root root       0 Jun 23 00:00 symbolpush.ssh_2014-06-23
-rw-r--r-- 1 root root       0 Jun 23 00:00 ldapproxy_client_cert_2014-06-23
-rw-r--r-- 1 root root       0 Jun 23 00:00 ldapproxy_2014-06-23
-rw-r--r-- 1 root root       0 Jun 23 00:00 db-internal-pg-rw.2014-06-23.log
-rw-r--r-- 1 root root 2289177 Jun 24 00:00 unified-http-to-https-redirect.2014-06-23.log
-rw-r--r-- 1 root root       0 Jun 24 00:00 ldapproxy_client_cert_2014-06-24
-rw-r--r-- 1 root root       0 Jun 24 00:00 symbolpush.ssh_2014-06-24
-rw-r--r-- 1 root root       0 Jun 24 00:00 ldapproxy_2014-06-24
-rw-r--r-- 1 root root       0 Jun 24 00:00 db-internal-pg-rw.2014-06-24.log
-rw-r--r-- 1 root root 4307488 Jun 25 00:00 unified-http-to-https-redirect.2014-06-24.log
-rw-r--r-- 1 root root       0 Jun 25 00:00 symbolpush.ssh_2014-06-25
-rw-r--r-- 1 root root       0 Jun 25 00:00 ldapproxy_client_cert_2014-06-25
-rw-r--r-- 1 root root       0 Jun 25 00:00 ldapproxy_2014-06-25
-rw-r--r-- 1 root root       0 Jun 25 00:00 db-internal-pg-rw.2014-06-25.log
-rw-r--r-- 1 root root 2212793 Jun 26 00:00 unified-http-to-https-redirect.2014-06-25.log
-rw-r--r-- 1 root root       0 Jun 26 00:00 symbolpush.ssh_2014-06-26
-rw-r--r-- 1 root root       0 Jun 26 00:00 ldapproxy_client_cert_2014-06-26
-rw-r--r-- 1 root root       0 Jun 26 00:00 db-internal-pg-rw.2014-06-26.log
-rw-r--r-- 1 root root       0 Jun 26 00:00 ldapproxy_2014-06-26
-rw-r--r-- 1 root root 1592115 Jun 26 16:39 unified-http-to-https-redirect.2014-06-26.log
-rw-r--r-- 1 root root       0 Jun 26 16:40 symbolpush-22.ssh_2014-06-26
[[e-mail]1 ~]#

Comment 3

[david garvey:dgarvey]

This looks like a related bug1043058.

Comment 4

[MOC Nagios API]

Automated alert recovery:

Hostname: metrics-logger1.private.scl3.mozilla.com
Service:  Check received log volume
State:    OK
Output:   OK: All log volumes are within normal for the last 24 hours