726384 - bm-vmware01.build.sjc1 Down

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Rick Bryce [:rbryce]]

Server started to alert ilo communication errors. I restarted the iLO card to hopefully regain access to Blade 7 the blade appears failed

Comment 1

[Rick Bryce [:rbryce]]

The blade seemed to crash as soon I as logged in after the iLO reset.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

That host was running the following VMs:

dev-stage1
moz2-linux-slave19
moz2-linux-slave27
moz2-linux-slave34
moz2-linux-slave37
moz2-linux-slave39
moz2-linux64-template
preproduction-master
preproduction-stage
try-linux-slave02
try-linux-slave03
try-linux-slave22

Of those, we can survive until working hours without the slaves.  That leaves

preproduction-master
preproduction-stage
dev-stage1

all of which are a part of the releng dev (aka preprod aka staging) environment, and not mission-critical, but they should be up sooner rather than later as releng can't stage changes without those components.  If possible, we should re-start the latter three VMs on another ESX host.  I don't know how to do that in vSphere, but I think it's possible..

Comment 3

[Rick Bryce [:rbryce]]

preproduction-master
preproduction-stage

Have been moved to other vm hosts.  Dev-stage1 is still in the process of copying files should finish up in an hour or so.

Comment 4

[Rick Bryce [:rbryce]]

The VMs listed above are running again.  Bm-vmware01 suddenly came back hours later.  I dont trust the hardware and am reluctant to bring the host back into the esx cluster until a hardware diagnostic can be run.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Of the VMs in comment #2, only preproduction-master and preproduction-stage are responding to ping. Could we have dev-stage1 back asap, and the others at your convenience ?

Comment 6

[Rick Bryce [:rbryce]]

:nthomas I forgot to change the MAC address in dhcp when I migrated the VM.  Server should be up now.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Thanks for fixing up dev-stage1. What's the plan for investigating the host problem and bringing up the remaining VMs ?

Comment 8

[Rick Bryce [:rbryce]]

I have tried to find with no success any problems with this server.

Comment 9

[Amy Rich [:arr] [:arich]]

rbeyce: can we bring it back up along with the linux builders vms?

Comment 10

[Rick Bryce [:rbryce]]

bm-vmware01 is back online.  I started the slaves below. I still have no clue what caused bm-vmware01 to act so badly for 12 hours and then suddenly resurrect. For now I think preproduction-master, preproduction-stage,and dev-stage1 should stay on other esx hosts.

--booted vms --
moz2-linux-slave19
moz2-linux-slave27
moz2-linux-slave34
moz2-linux-slave37
moz2-linux-slave39
moz2-linux64-template
try-linux-slave02
try-linux-slave03
try-linux-slave22
--