765493 - esx7.private.scl3 crashed, rebooted

Status: RESOLVED FIXED

(Infrastructure & Operations :: Virtualization, task)

Description

[Michael Burns [:mburns]]

Attachment: esx7 purple screen of death

[06:40:37] <nagios-scl3> [544] vc1.private.scl3.mozilla.com:vmware_vcenter is CRITICAL: [esx7.private.scl3.mozilla.com] Host connection and power state is red BR [esx7.private.scl3.mozilla.com] vSphere HA host status is red


esx7 crashed around 6:40am PST. Lerxst gave the all-clear to reboot it at 9:00am. Appears to be the same 'motherboard interupt' problem of previous hard-lockups.

Comment 1

[Dan Parsons [:lerxst]]

Greg, please open a SR with VMware for this. We've seen this error many times before but never on the same host twice.

Comment 2

[Greg Cox [:gcox]]

SR 12187737806 opened, target response is mid/late on the 19th.

Comment 3

[Greg Cox [:gcox]]

VMware got back with me.  LINT1 is an NMI from hardware (which we knew).  Per (many directions) we were suggested to upgrade firmware, which I've done in the SCL3 batch (PHX1 and HCI are untouched right now allowing for settle-out time).  Upgraded stuff included onboard and HP nics, iLO, and the SmartArray.

According to VMware, HP wants people to get up with them on LINT1's(!?).  Working on getting an account set up to case this with them.

Comment 4

[Greg Cox [:gcox]]

Upgraded PHX1's firmwares.  HCI untouched.

Opened HP case 4640647448; gathered requested logs.

Comment 5

[Greg Cox [:gcox]]

The motherboard got replaced on 28 June.  1.5 passes of memtest came back clean.  Going to do more burning over the weekend.

Comment 6

[Greg Cox [:gcox]]

After the mobo swap, ran the box on a CPU burn for just under 3 days, no issues.  Booted to ESX, ran at full CPU and 75% memory for a day, no issues.

Having not seen anything for that long, calling it.  Returned to service.