1108314 - Check received log volume on metrics-logger1.private.scl3.mozilla.com is CRITICAL: CRITICAL: Daily log size from /data/stats/logs/zlb1.ops.pek1.mozilla.com is low at 46MB compared to running average 54MB

Status: RESOLVED FIXED

(Infrastructure & Operations :: MOC: Problems, task)

Description

[MOC Nagios API]

Automated alert report from nagios1.private.scl3.mozilla.com:

Hostname: metrics-logger1.private.scl3.mozilla.com
Service:  Check received log volume
State:    CRITICAL
Output:   CRITICAL: Daily log size from /data/stats/logs/zlb1.ops.pek1.mozilla.com is low at 46MB compared to running average 54MB

Runbook:  http://m.allizom.org/Check+received+log+volume

Comment 1

[david garvey:dgarvey]

Humm, I don't know why one would be doing sudo as root?

I would do something this...
su -c "rsync -av '--include=*.gz' '--exclude=*' --remove-source-files -e 'ssh -o StrictHostKeyChecking=no' /var/log/zeus/ [e-mail]:/data/stats" logpush

[[e-mail]1 ~]# bash -x /usr/local/bin/rsync_files.sh
+ LOCKFILE=/tmp/rsync.lock
++ hostname
+ HOSTNAME=zlb1.ops.pek1.mozilla.com
+ DIR=/var/log/zeus
+ LOGSERVER=metrics-logger1.private.scl3.mozilla.com
+ create_lock /tmp/rsync.lock
+ LOCK_NAME=/tmp/rsync.lock
+ [[ -e /tmp/rsync.lock ]]
+ echo 22158
+ return 0
+ cd /var/log/zeus
+ find /var/log/zeus -type f -name '*.access*' -mmin +30 '!' -name '*gz' -print0
+ xargs -0 -r -P 2 -n 1 nice -n 10 gzip
+ find /var/log/zeus -type f -name '*.access*.gz' -print0
+ xargs -0 -r chown logpull
+ RESULTLOGDIR=/var/log/rsync
+ '[' '!' -d /var/log/rsync ']'
++ date +%Y-%m-%d-%H
+ RESULTLOG=/var/log/rsync/rsync-metrics-logger1.private.scl3.mozilla.com-2014-12-07-00
+ sudo -u logpull rsync -av '--include=*.gz' '--exclude=*' --remove-source-files -e 'ssh -o StrictHostKeyChecking=no' /var/log/zeus/ [e-mail]:/data/stats/logs/zlb1.ops.pek1.mozilla.com/
+ '[' 0 -ne 0 ']'
+ rm -f /tmp/rsync.lock
[[e-mail]1 ~]# crontab -l | grep rsync
45 * * * * /usr/local/bin/rsync_files.sh
[[e-mail]1 ~]#

Comment 2

[david garvey:dgarvey]

But in any case this doesn't have any files. Must be a slow night in china?

[[e-mail]1 ~]# find /var/log/zeus -type f -name '*.access*' -mmin +30 '!' -name '*gz' -print0
[[e-mail]1 ~]#

Comment 3

[MOC Nagios API]

Automated alert recovery:

Hostname: metrics-logger1.private.scl3.mozilla.com
Service:  Check received log volume
State:    OK
Output:   OK: All log volumes are within normal for the last 24 hours