RAM problem on bm-vmware13

RESOLVED FIXED

Status

--
critical
RESOLVED FIXED
10 years ago
4 years ago

People

(Reporter: nthomas, Assigned: phong)

Tracking

Details

(Whiteboard: HP case 3604400561)

(Reporter)

Description

10 years ago
Nagios said at 1:10 PST today
  [69] bm-vmware13.build:health is CRITICAL: CRITICAL - dimm module 6 @ cartridge 0 needs attention (dimm is degraded)
and 5 minutes later
  [72] bm-vmware13.build:hplog is WARNING: WARNING 0000: Corrected Memory Error threshold exceeded (System Memory, Memory Module 6)

Could we check what proportion of memory this affects asap ?
(Assignee)

Updated

10 years ago
Assignee: server-ops → phong
(Assignee)

Comment 1

10 years ago
HP Case ID 3604400561
(Reporter)

Comment 2

10 years ago
Is the hardware (or ESX) handling this gracefully, or are we in a situation where we might be producing bogus builds and confusing developers ? Does the load-balancing automatically notice the memory restriction and transition VMs to other hosts ?
(Assignee)

Comment 3

10 years ago
This is a hardware problem.  I have a replacement on order to replace the bad DIMM.  I will migrate all the VM's off that host and shutdown the ESX server.
Whiteboard: HP case 3604400561
(Assignee)

Comment 4

10 years ago
Bad DIMM replaced and system is back to normal.  Putting ESX host back into the build pool.
Status: NEW → RESOLVED
Last Resolved: 10 years ago
Resolution: --- → FIXED
(Reporter)

Comment 5

10 years ago
Thanks for the quick turnaround.
(Reporter)

Comment 6

10 years ago
Just got this again
[43] bm-vmware13.build:hplog is WARNING: WARNING 0000: Corrected Memory Error threshold exceeded (System Memory, Memory Module 6)

:-(
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 7

10 years ago
That was an old alert that didn't get cleared.
Status: REOPENED → RESOLVED
Last Resolved: 10 years ago10 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.