507011 - please dig through the nagios logs

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[[e-mail] (:bhearsum)]

We had an issue with staging-master.build.mozilla.org around 4am PDT on Sunday - can you look through the logs and see if there's anything there?

Same for production-master around 11:15am PDT today (Tuesday) - anything there?

Comment 1

[Shyam Mani [:fox2mike]]

Nothing on staging on 25,26,27,28 (results from 26,27 and 28 are pasted below). Looks like the the host itself had no issues.

[root@dm-nagios01 archives]# grep staging-master.build nagios-07-26-2009-00.log | grep -v EXTERNAL | grep -v PASSIVE 
[1248505200] CURRENT HOST STATE: staging-master.build;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 132.25 ms
[1248505200] CURRENT SERVICE STATE: staging-master.build;PING;OK;HARD;1;PING OK - Packet loss = 0%, RTA = 0.52 ms
[1248505200] CURRENT SERVICE STATE: staging-master.build;avg load;OK;HARD;1;OK - load average: 0.12, 0.12, 0.18
[1248505200] CURRENT SERVICE STATE: staging-master.build;buildbot;WARNING;HARD;4;PROCS WARNING: 0 processes with command name buildbot
[1248505200] CURRENT SERVICE STATE: staging-master.build;disk - /builds;OK;HARD;1;DISK OK - free space: /builds 22302 MB (77% inode=90%):
[1248505200] CURRENT SERVICE STATE: staging-master.build;processes;OK;HARD;1;PROCS OK: 32 processes with STATE = RSZDT
[1248505200] CURRENT SERVICE STATE: staging-master.build;root partition;OK;HARD;1;DISK OK - free space: / 4413 MB (55% inode=92%):
[1248505200] CURRENT SERVICE STATE: staging-master.build;users;OK;HARD;1;USERS OK - 0 users currently logged in

[root@dm-nagios01 archives]# grep staging-master.build nagios-07-27-2009-00.log | grep -v EXTERNAL | grep -v PASSIVE 
[1248591600] CURRENT HOST STATE: staging-master.build;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 4.95 ms
[1248591600] CURRENT SERVICE STATE: staging-master.build;PING;OK;HARD;1;PING OK - Packet loss = 0%, RTA = 2.09 ms
[1248591600] CURRENT SERVICE STATE: staging-master.build;avg load;OK;HARD;1;OK - load average: 0.18, 0.25, 0.26
[1248591600] CURRENT SERVICE STATE: staging-master.build;buildbot;WARNING;HARD;4;PROCS WARNING: 0 processes with command name buildbot
[1248591600] CURRENT SERVICE STATE: staging-master.build;disk - /builds;OK;HARD;1;DISK OK - free space: /builds 21969 MB (76% inode=90%):
[1248591600] CURRENT SERVICE STATE: staging-master.build;processes;OK;HARD;1;PROCS OK: 32 processes with STATE = RSZDT
[1248591600] CURRENT SERVICE STATE: staging-master.build;root partition;OK;HARD;1;DISK OK - free space: / 4411 MB (55% inode=92%):
[1248591600] CURRENT SERVICE STATE: staging-master.build;users;OK;HARD;1;USERS OK - 0 users currently logged in

[root@dm-nagios01 archives]# grep staging-master.build nagios-07-28-2009-00.log | grep -v EXTERNAL | grep -v PASSIVE 
[1248678000] CURRENT HOST STATE: staging-master.build;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 1.22 ms
[1248678000] CURRENT SERVICE STATE: staging-master.build;PING;OK;HARD;1;PING OK - Packet loss = 0%, RTA = 0.00 ms
[1248678000] CURRENT SERVICE STATE: staging-master.build;avg load;OK;HARD;1;OK - load average: 0.01, 0.11, 0.10
[1248678000] CURRENT SERVICE STATE: staging-master.build;buildbot;WARNING;HARD;4;PROCS WARNING: 0 processes with command name buildbot
[1248678000] CURRENT SERVICE STATE: staging-master.build;disk - /builds;OK;HARD;1;DISK OK - free space: /builds 21909 MB (76% inode=89%):
[1248678000] CURRENT SERVICE STATE: staging-master.build;processes;OK;HARD;1;PROCS OK: 33 processes with STATE = RSZDT
[1248678000] CURRENT SERVICE STATE: staging-master.build;root partition;OK;HARD;1;DISK OK - free space: / 4408 MB (55% inode=92%):
[1248678000] CURRENT SERVICE STATE: staging-master.build;users;OK;HARD;1;USERS OK - 1 users currently logged in

Comment 2

[Shyam Mani [:fox2mike]]

And nothing on production as well :

[root@dm-nagios01 nagios]# grep production-master.build nagios.log | grep -v EXTERNAL | grep -v PASSIVE 
[1248764400] CURRENT HOST STATE: production-master.build;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 27.19 ms
[1248764400] CURRENT SERVICE STATE: production-master.build;PING;OK;HARD;1;PING OK - Packet loss = 0%, RTA = 0.02 ms
[1248764400] CURRENT SERVICE STATE: production-master.build;avg load;OK;HARD;1;OK - load average: 0.84, 0.79, 0.60
[1248764400] CURRENT SERVICE STATE: production-master.build;buildbot;OK;HARD;1;PROCS OK: 2 processes with command name buildbot
[1248764400] CURRENT SERVICE STATE: production-master.build;disk - /builds;OK;HARD;1;DISK OK - free space: /builds 35684 MB (74% inode=87%):
[1248764400] CURRENT SERVICE STATE: production-master.build;root partition;OK;HARD;1;DISK OK - free space: / 2657 MB (33% inode=89%):
[1248805408] SERVICE ALERT: production-master.build;buildbot;WARNING;SOFT;1;PROCS WARNING: 1 process with command name buildbot
[1248805588] SERVICE ALERT: production-master.build;buildbot;OK;SOFT;2;PROCS OK: 2 processes with command name buildbot

Checked the previous day too, nothing abnormal. 

dmoore, just FYI, I checked the logs in /var/log/nagios/archives and /var/log/nagios on dm-nagios01, I'm pretty sure that's where the stuff is?

Comment 3

[[e-mail] (:bhearsum)]

Alright, thanks for having a look.