448908 - bm-vmware05 crashed, taking down a bunch of VMs

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Armen [:armenzg]]

I tried to ssh or vnc to it but no luck

Any suggestion?

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

There's a lot of this on the console:
  sd 0:0:0:0 timit out command, waited 360s
  end_request: I/O error, dev sda, sector 20291311
for assorted sectors. And some 
  Read-error on swap-device (0:0:0:20291319)
but not always paired with the first set. It's not using very much CPU, RAM, or doing a bunch of disk or network access, according to the Performance info in the VI Client.

Can't get a prompt on the console, so trying rebooting.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

The gentle method (using "Reboot Guest VM") got no response. The more determined method (using "Reset") is hung at 95% completion, so need help from Server Ops. There isn't a good place to reclone this machine from so lets try to recover it first.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

FWIW, looks like it failed sometime after 09:10 PDT on Wed 30 Jul.

Comment 4

[matthew zeier [:mrz]]

management process on bm-vmware05 died, trying to figure out how to restart.

Comment 5

[matthew zeier [:mrz]]

Everything is pointing to an iSCSI issue - can't tell which iSCSI LUN was at fault.

While debugging with VMware on the phone, the box crashed taking down the following VMs:

fx-linux-1.9-slave1
moz2-win32-slave06
prometheus-vm
tb-linux-tbox
try-master

Punting over to RE to bring things back up.  Still working with VMware on root cause.

Comment 6

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Talked to mrz:

1) Filed bug#449059 to track reviving the downed VMs.
2) Pushing this bug back to IT to track fixing the root cause of kernel panic on bm-vmware05.

Comment 7

[matthew zeier [:mrz]]

vmware blades HP's management tools.  HP doesn't think so but does recommend upgrading from 8.0 to 8.1.  

I doubt it's related to that since it only crashed after the vmware tech was trying to fix some other SAN issues but I'll upgrade.

Comment 8

[matthew zeier [:mrz]]

updated.