1340352 - Autophone - autophone-2 is down

Status: RESOLVED FIXED

(Testing Graveyard :: Autophone, defect)

Description

[Bob Clary [:bc] (inactive)]

autophone-2 experienced a set of disconnections and reboots after the new hub was installed and after attempting to reboot at 4:03PM has failed to come back up.

Van, is there any way you can take a look at it before Monday?

uber, if Van can't make it will you be in the office on Friday? I'd rather not leave it in what ever state it is in over the weekend.

If we can get it to boot back up, I would like to remove the hub and connect the devices back to the host directly.

If we can't get it to boot back up, I would like to power it down until we can investigate whether we need to have it serviced.

Comment 1

[Bob Clary [:bc] (inactive)]

Timeline of disconnections and reboots since new hubs added:

Inventory https://docs.google.com/spreadsheets/d/1ppMtFUS6by42QCv_b_6EPQko30TaswAxrmFLb-FxHcI/edit?ts=58a61d0d#gid=0

14:22 autophone-2 process finished starting
14:41 autophone-2 nexus-6p-12 disconnected   USB3HUB2.5
14:46 autophone-2 rebooted
14:54 autophone-2 nexus-4-8 failed to init   USB3HUB2.2
14:59 autophone-2 nexus-4-9 failed to init   USB3HUB2.3
15:04 autophone-2 nexus-5-1 failed to init   USBCARD3.1
15:09 autophone-2 nexus-5-8 failed to init   USB2.1
15:14 autophone-2 nexus-6-1 failed to init   USBCARD3.3
15:19 autophone-2 nexus-6p-7 failed to init  USB3HUB2.4
15:24 autophone-2 nexus-6p-12 failed to init USB3HUB2.5
15:29 autophone-2 nexus-6p-13 failed to init USBCARD3.2
15:34 autophone-2 nexus-9-1 failed to init   USB3HUB2.1
15:34 autophone-2 process finished starting
15:44 autophone-2 nexus-5-3 disconnected     USB2.4
16:03 autophone-2 rebooted
      autophone-2 offline

14:10 autophone-1 process finished starting
23:45 autophone-1 nexus-6p-11 disconnected   USB3HUB2.2
23:49 autophone-1 rebooted
23:53 autophone-1 process finished starting

14:10 autophone-3 process finished starting
14:48 autophone-3 pixel-02 disconnected      USB3HUB2.1
15:07 autophone-3 rebooted
15:12 autophone-3 process finished starting
18:49 autophone-3 pixel-02 disconnected      USB3HUB2.1
19:08 autophone-3 rebooted
19:13 autophone-3 process finished starting
20:33 autophone-3 pixel-02 disconnected      USB3HUB2.1
22:52 autophone-3 rebooted
22:56 autophone-3 process finished starting

No further issues past 22:56

Comment 2

[Bob Clary [:bc] (inactive)]

2017-02-17
02:55 autophone-3 pixel-02 disconnected      USB3HUB2.1
03:14 autophone-3 rebooted
03:18 autophone-3 process finished starting

Comment 3

[Bob Clary [:bc] (inactive)]

06:31 autophone-1 nexus-6p-11 disconnected   USB3HUB2.2
06:35 autophone-1 rebooted
06:39 autophone-1 process finished starting

Comment 4

[Bob Clary [:bc] (inactive)]

08:25 autophone-3 pixel-02 disconnected      USB3HUB2.1
08:44 autophone-3 rebooted
08:48 autophone-3 process finished starting

Comment 5

[Bob Clary [:bc] (inactive)]

10:00 autophone-3 pixel-02 disconnected      USB3HUB2.1
10:19 autophone-3 rebooted
10:23 autophone-3 process finished starting

Comment 6

[Bob Clary [:bc] (inactive)]

12:04 autophone-3 pixel-02 disconnected      USB3HUB2.1
12:23 autophone-3 rebooted
12:27 autophone-3 process finished starting

I removed pixel-02 from the manifests to prevent this from reoccurring until we can fix the issue.

https://github.com/mozilla/autophone/commit/f36f7948d5580d7e1dbf85a69f598e2e68421e50

Comment 7

[Van Le [:van]]

brought autophone2 back online and worked with :bc to move some phones around (especially off the hub connected to autophone2).

Comment 8

[Bob Clary [:bc] (inactive)]

With Van's help autophone-2 was rebooted. It appeared to have hung. We've moved most devices off of the hubs until we have a handle on the problem with autophone-2.

nexus-6p-7, nexus-6p-12 moved from autophone-2 to autophone-3
pixel-02                moved from autophone-3 to autophone-2

deployed
https://github.com/mozilla/autophone/commit/4df2d5a533e31a9ecee5b0e50f6e50db0eeaf3eb

and rebooted everyone. nexus-6p-4 on autophone-3 (twitter-remote on mozilla-inbound, autoland) is missing but hopefully we can get it to come back to us.

We'll revisit the hubs when we have more time.

Thanks Van!