472183 - RAM problem on bm-vmware13

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Nagios said at 1:10 PST today
  [69] bm-vmware13.build:health is CRITICAL: CRITICAL - dimm module 6 @ cartridge 0 needs attention (dimm is degraded)
and 5 minutes later
  [72] bm-vmware13.build:hplog is WARNING: WARNING 0000: Corrected Memory Error threshold exceeded (System Memory, Memory Module 6)

Could we check what proportion of memory this affects asap ?

Comment 1

[Phong Tran [:phong]]

HP Case ID 3604400561

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Is the hardware (or ESX) handling this gracefully, or are we in a situation where we might be producing bogus builds and confusing developers ? Does the load-balancing automatically notice the memory restriction and transition VMs to other hosts ?

Comment 3

[Phong Tran [:phong]]

This is a hardware problem.  I have a replacement on order to replace the bad DIMM.  I will migrate all the VM's off that host and shutdown the ESX server.

Comment 4

[Phong Tran [:phong]]

Bad DIMM replaced and system is back to normal.  Putting ESX host back into the build pool.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Thanks for the quick turnaround.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Just got this again
[43] bm-vmware13.build:hplog is WARNING: WARNING 0000: Corrected Memory Error threshold exceeded (System Memory, Memory Module 6)

:-(

Comment 7

[Phong Tran [:phong]]

That was an old alert that didn't get cleared.