482715 - build esx hosts/VMs having network problems

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[[e-mail] (:bhearsum)]

Awhile ago Nagios reported some Linux machines being unresponsive. Indeed, they were. When I tried to reset them those requests never completed, erroring with "An error occurred while communicating with the remote host" or "Operation timed out.".

Shortly afterwards two of the ESX hosts showed up as "Disconnect", bm-vmware04 and bm-vmware10. This has affected numerous VMs, including some production moz2-* slaves and the production try server master.

This is a tracking bug for getting all of this up and running again.

Comment 1

[Philippe M. Chiasson (:gozer)]

This has most likely taken down tb-linux-tbox.build (TB linux builder) along for the ride

Comment 2

[[e-mail] (:bhearsum)]

Yeah, looks like you're right gozer. That's worrying, because it's not one of the "disconnected" VMs.

Comment 3

[[e-mail] (:bhearsum)]

OK. After much swearing almost everything is back up. moz2-linux-slave13 and tb-linux-tbox are still in a weird state. Mrz tells me he needs to contact VMware to figure out what's going on.

Comment 4

[Aravind Gottipati [:aravind]]

The problem happened because I restarted the network interfaces on the eql array to clear things up from a previous problem we had.  Specifially, I wanted to ensure that it had flowcontrol turned on.  Not only did it not help the other problem, but caused this build outage (My apologies).  Also, mrz still has an open case with vmware about the two vms that won't play nice.

Also, they said that there should be some timeout settings in the VMs themselves that should make them immune to such failures (for up to a minute).  I unfortunately don't know what those settings are.  Will keep this bug open until we can figure out how to get those two remaining VMs back on track.

Comment 5

[matthew zeier [:mrz]]

Two updates:

bm-vmware08's service console (think "Linux shell") doesn't have enough memory provisioned and is swapping.  The hostd process (the one that talks to VC) is running very slow.  It sometimes drops out of VC because of that.

I've bumped it up to 512MB but that requires a reboot.  Need to schedule a time to put this host into maintenance mode and get all the VMs off to reboot.  Normally I'd just do this but I haven't any idea how successful that will be.

VMs affected:
moz2-linux-slave13
staging-prometheus-vm02
moz2-linux0slave05
staging-crazyhorse
staging-patrolcles
moz-win32-slave20
moz-win32-slave23

bm-vmware05 is failing to get a lock on tb-linux-tbox's VM and hostd there is simply hung when trying to query that VM.  This host needs a reboot to fix.

VMs affected:
karma
crazhorse
fx-win32-1.9-slave2
tb-linux-tbox
fx-linux-1.9-slave08
bm-l10n-centos5-01
production-prometheus-vm
fx-win32-tbox

Comment 6

[matthew zeier [:mrz]]

btw, the VMs are fine (except the two known ones) but are uncontrollable from VI/VC.  So you could put this off as long as you want as long as you don't have to manage them .

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #5)
> VMs affected (bm-vmware08):
> staging-prometheus-vm02
> staging-crazyhorse
> staging-patrocles

All off right now.

> moz2-linux-slave13
> moz2-linux-slave05
> moz-win32-slave20
> moz-win32-slave23

We can take these out of the pool any time they're idle. Give us a ping to let us know when you're available and we can do that.

> VMs affected (bm-vmware05):
> fx-win32-1.9-slave2

Idle between releases, next one starts ~ March 30.

> karma

Fx/Tb 2.0.0.x nightly l10n builds, can interrupt these any time.

> crazyhorse

Tb 2.0.0.x nightly/hourly builds, can interrupt this any time (although we are 1 week out from 2.0.0.21 getting release, so it's possible there might be a respin).

> fx-linux-1.9-slave08

Fx3.0 unit tests, one of three machines so we can take it offline whenever.

> bm-l10n-centos5-01

Idle.

> production-prometheus-vm

Idle now Fx2 releases are done.

> fx-win32-tbox

Fx3.0 nightly/hourly builds, short outage (~1 hour) would be OK but longer ones blocked by code freeze for 3.0.8 on March 17.

> tb-linux-tbox

Coordinate with gozer.

Comment 8

[Philippe M. Chiasson (:gozer)]

(In reply to comment #7)
> (In reply to comment #5)
> > VMs affected (bm-vmware08):
> [...]
> > tb-linux-tbox
> 
> Coordinate with gozer.

No need for coordination here, as that VM is down/dead right now, anything that will bring it back up and running is fine with me, anytime.

Comment 9

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #4)
> The problem happened because I restarted the network interfaces on the eql
> array to clear things up from a previous problem we had.  Specifially, I wanted
> to ensure that it had flowcontrol turned on.  Not only did it not help the
> other problem, but caused this build outage (My apologies).  
Any details on what the "other problem" is? Is it detailed in a bug someplace?


> Also, they said that there should be some timeout settings in the VMs
> themselves that should make them immune to such failures (for up to a minute). 
> I unfortunately don't know what those settings are.  
Can you get those settings from VMware? We can try them in staging VMs first to make sure they dont have any unhappy sideeffects.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

I've shutdown everything on bm-vmware05 as best I can, so go ahead with rebooting that one.

Comment 11

[matthew zeier [:mrz]]

05's rebooted

Comment 12

[matthew zeier [:mrz]]

08's been rebooted and back up.

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

I think everything is back up now.

Comment 14

[Philippe M. Chiasson (:gozer)]

tb-linux-tbox back in business! Thanks.

Comment 15

[matthew zeier [:mrz]]

Resolving.