715478 - surf.m.o has badd DIMM and is missing hardware monitoring in nagios

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Justin Dow [:jabba]]

Looks like we are missing the hplog and the hpasm checks in nagios, both of which might have alerted about the bad memory. I caught it via an HP Insights Management Trap alarm:

Trap-ID=6064

correctable \ uncorrectable  Errors detected.

and whent investigating on the box. Running nrpe commands manually:

[root@surf ~]# /usr/lib/nagios/plugins/custom/check_hpasm -t 20
CRITICAL - dimm module 2 @ cartridge 0 needs attention (dimm is degraded)

Comment 1

[Amy Rich [:arr] [:arich]]

I've added the two HP checks to surf.

Comment 2

[Amy Rich [:arr] [:arich]]

coop, joduinn, we should do this at the same time as the fsck in bug 715026 since they both require surf to be out of commission.  Phong, do we have the memory available to put in the machine next week (when cshields is trying to schedule a tree closure to upgrade hg)?

Comment 3

[Phong Tran [:phong]]

I also saw that one of the drive is dead.  I replaced the 72 GB 10K ULTRA320 spare.

I have a spare RAM module.

Comment 4

[Rick Bryce [:rbryce]]

Ram in dimm slot 2 replaced.

Comment 5

[Rick Bryce [:rbryce]]

Reopened -- Still need to add nagios checks or confirm they are in place.

Comment 6

[Amy Rich [:arr] [:arich]]

I added the checks back in comment 1.