Closed Bug 1043831 Opened 10 years ago Closed 10 years ago

Check received log volume on metrics-logger1.private.scl3.mozilla.com is CRITICAL: CRITICAL: Daily log size from /data/stats/logs/zlb11.ops.phx1.mozilla.com is low at 0MB compared to running average 1215MB

Categories

(mozilla.org Graveyard :: Server Operations: MOC, task)

Product:
mozilla.org Graveyard
Component:
Server Operations: MOC
Platform:
Other
Other
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: nagiosapi, Assigned: dgarvey)

References

(
URL
)

Details

(Whiteboard: [id=nagios1.private.scl3.mozilla.com:388561]) 

Automated alert report from nagios1.private.scl3.mozilla.com:

Hostname: metrics-logger1.private.scl3.mozilla.com
Service:  Check received log volume
State:    CRITICAL
Output:   CRITICAL: Daily log size from /data/stats/logs/zlb11.ops.phx1.mozilla.com is low at 0MB compared to running average 1215MB

Runbook:  http://m.allizom.org/Check+received+log+volume
I am not sure how the logs are pulled here? I don't think pull is not working?

[root@metrics-logger1.private.scl3 ~]# ls -lrt /data/stats/logs/zlb11.ops.phx1.mozilla.com | tail -3
-rw-r--r-- 1 logpull logpull   5726635 Jul 23 17:30 snippets.mozilla.com.access_2014-07-23-17.gz
-rw-r--r-- 1 logpull logpull     40406 Jul 23 17:30 input.mozilla.org.access_2014-07-23-17.gz
-rw-r--r-- 1 logpull logpull   6204458 Jul 23 17:30 download.mozilla.org.access_2014-07-23-17.gz
[root@metrics-logger1.private.scl3 ~]# find /data/stats/logs/zlb11.ops.phx1.mozilla.com -type f -mtime -1
[root@metrics-logger1.private.scl3 ~]#
Assignee: nobody → dgarvey
Not sure how the logs are supposed to get here?

[dgarvey@metrics-logger1.private.scl3 ~]$ ls -lrt /data/stats/logs/zlb11.ops.phx1.mozilla.com | tail
-rw-r--r-- 1 logpull logpull       234 Jul 23 17:27 bugzilla.mozilla.org.access_2014-07-23-17.gz
-rw-r--r-- 1 logpull logpull       585 Jul 23 17:29 aus3.mozilla.org.access_2014-07-23-17.gz
-rw-r--r-- 1 logpull logpull       869 Jul 23 17:30 snippets-stats.mozilla.org.access_2014-07-23-17.gz
-rw-r--r-- 1 logpull logpull       264 Jul 23 17:30 download-stats.mozilla.org.access_2014-07-23-17.gz
-rw-r--r-- 1 logpull logpull       225 Jul 23 17:30 bugzilla.mozilla.org-http.access_2014-07-23-17.gz
-rw-r--r-- 1 logpull logpull  21330926 Jul 23 17:30 www.mozilla.org.access_2014-07-23-17.gz
-rw-r--r-- 1 logpull logpull    192591 Jul 23 17:30 support.mozilla.com.access_2014-07-23-17.gz
-rw-r--r-- 1 logpull logpull   5726635 Jul 23 17:30 snippets.mozilla.com.access_2014-07-23-17.gz
-rw-r--r-- 1 logpull logpull     40406 Jul 23 17:30 input.mozilla.org.access_2014-07-23-17.gz
-rw-r--r-- 1 logpull logpull   6204458 Jul 23 17:30 download.mozilla.org.access_2014-07-23-17.gz
[dgarvey@metrics-logger1.private.scl3 ~]$
:dgarvey - I think this is related to Bug 1043058, where zlb11 was pulled out of the cluster.
Depends on: 1043058
Automated alert recovery:

Hostname: metrics-logger1.private.scl3.mozilla.com
Service:  Check received log volume
State:    OK
Output:   OK: All log volumes are within normal for the last 24 hours
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.