437049 - bm-vmware05 is "not responding" according to the VI Client

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[[e-mail] (:bhearsum)]

This host has a few tier1 machines on it, filing as blocker.

Comment 1

[[e-mail] (:bhearsum)]

Just realized that even though the host is connected, the tier1 machines are still reachable. downgrading

Comment 2

[matthew zeier [:mrz]]

Looks removed from VC too.  

This happened once before on another ESX host. The VC process on the ESX host was wedged and needed a restart.  For posterity, this is done by:

service  mgmt-vmware  stop
service  vmware-vpxa stop

service  mgmt-vmware  start
service  vmware-vpxa start

I will attempt to do that now!

Comment 3

[Justin Fitzhugh]

already tried that, removed it from VI and can't re-add it.

Comment 4

[matthew zeier [:mrz]]

bm-vmware04 refuses to see or re-discover netapp-b's LUNs.  This mostly affects fx-linux64-slave1 - it's running but the host can't find it's datastore (so very likely something is not working on the VM).  

I'm trying to move the existing VMs off bm-vmare05 so I can reboot it.  It's slower than usual probably because of netapp-b's missing LUNs.  I can either go at this (slowly) or take a quicker route and reboot the whole ESX host taking 5 VMs down with it.

Currently going on the slow path...

Comment 5

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Currently shutting down 4 VMs (try1-win32-slave, prometheus-vm, tb-linux-tbox, try1-linux-slave) on bm-vmware04. Managed to move "egg" to bm-vmware02.

Comment 6

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #5)
> Currently shutting down 4 VMs (try1-win32-slave, prometheus-vm, tb-linux-tbox,
> try1-linux-slave) on bm-vmware04. Managed to move "egg" to bm-vmware02.
> 
...errr...typo. That should be bm-vmware05, not bm-vmware04.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

We started the VMs at 3pm PDT after mrz got the host rebooted and back in the VI. prometheus-vm had some disk corruption and I had to restore some parts of perl from staging-prometheus-vm after fsck was finished. Asking rpm to verify all the installed packages gave only config changes after that.

I see that there were 4 attempts to move prometheus-vm to different storage and it's off now. So what's the story with this VM ? Can I turn it back on and provide updates to our nightly testers ? And what's the overall status of the host ?

Comment 8

[matthew zeier [:mrz]]

this was resolved yesterday and is related to the larger netapp perf issues.