1396401 - AWFY - Worker machines down 2017-09-03

Status: RESOLVED FIXED

(Testing Graveyard :: AWFY, enhancement)

Description

[Bob Clary [:bc] (inactive)]

We began receiving messages from the automatic alerts that the arewefastyet machines were not reporting results to the server.

It appears the last reports for the affected machines were at approximately 2017-09-01 22:20 PDT

I attempted to access the machines and could not reach

quantum-flow-reference-1.corp.mtv2.mozilla.com
awfy-win8.mtv2.mozilla.com
awfy-cmp-linux.corp.mtv2.mozilla.com
awfy-shells-macpro.corp.mtv2.mozilla.com

I was able to reach awfy-cmp-osx.corp.mtv2.mozilla.com which appeared to have been rebooted. I restarted the worker task.

I was also able to ssh and vnc/rdp into awfy-cmp-win.corp.mtv2.mozilla.com though the vnc/rdp sessions where not responsive. I believe I accidentally put this machine into hibernation during an attempt to reboot it.

My best guess is that there was a power interruption in the Mountain View office and not all of the machines are configured to power on after a power outage. This corresponds to the same time I began seeing networking problems with some of Autophone's devices but I did not notice a power outage causing the Autophone servers to reboot.

The follow machines need intervention from someone on site in the Mountain View office:

Near sstangl's desk:

quantum-flow-reference-1.corp.mtv2.mozilla.com
awfy-win8.mtv2.mozilla.com
awfy-shells-macpro.corp.mtv2.mozilla.com

In the IT Server room in Mountain View:

awfy-cmp-linux.corp.mtv2.mozilla.com
awfy-cmp-win.corp.mtv2.mozilla.com

Sean, Henri: Can you check on these machines when you come in on Tuesday ?

Comment 1

[Bob Clary [:bc] (inactive)]

Everyone except awfy-cmp-win.corp.mtv2.mozilla.com is up. I don't know why but we still need action on awfy-cmp-win.corp.mtv2.mozilla.com.

Comment 2

[Bob Clary [:bc] (inactive)]

hlangi powered awfy-cmp-win.corp.mtv2.mozilla.com back on and we are done.