1488022 - t-yosemite-r7-451 is unreachable

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Hardware, task)

Description

[Radu Iman[:riman]]

This worker is missing from Taskcluster. SSH connection is not working:
Stdio forwarding request failed: Session open refused by peer
ssh_exchange_identification: Connection closed by remote host

The last logs from Papertrail:

Aug 31 05:09:14 t-yosemite-r7-451.test.releng.mdc1.mozilla.com generic-worker: 2018/08/31 05:09:14 Resolved 1 tasks in total so far.
Aug 31 05:09:14 t-yosemite-r7-451.test.releng.mdc1.mozilla.com generic-worker: 2018/08/31 05:09:14 Completed all task(s) (number of tasks to run = 1)
Aug 31 05:09:14 t-yosemite-r7-451.test.releng.mdc1.mozilla.com generic-worker: 2018/08/31 05:09:14 Saving file file-caches.json (absolute path: /Users/cltbld/file-caches.json)
Aug 31 05:09:14 t-yosemite-r7-451.test.releng.mdc1.mozilla.com generic-worker: 2018/08/31 05:09:14 Saving file directory-caches.json (absolute path: /Users/cltbld/directory-caches.json)
Aug 31 05:09:14 t-yosemite-r7-451.test.releng.mdc1.mozilla.com generic-worker: 2018/08/31 05:09:14 Exiting worker with exit code 0
Aug 31 05:09:14 t-yosemite-r7-451.test.releng.mdc1.mozilla.com sudo:   cltbld : TTY=unknown ; PWD=/Users/cltbld ; USER=root ; COMMAND=/sbin/reboot
Aug 31 05:09:14 t-yosemite-r7-451.test.releng.mdc1.mozilla.com reboot: rebooted by cltbld
Aug 31 05:09:14 t-yosemite-r7-451.test.releng.mdc1.mozilla.com reboot: SHUTDOWN_TIME: 1535717354 827360

Comment 1

[Zsolt Fay [:zfay]]

It also isn't pingable:

PING t-yosemite-r7-451.test.releng.mdc1.mozilla.com (10.49.56.75) 56(84) bytes of data.
^C
--- t-yosemite-r7-451.test.releng.mdc1.mozilla.com ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3060ms

Dave, could you look deeper into this one, please?

Comment 2

[:dhouse]

I powered it off and then back on manually with snmp, and it does not respond to ping or ssh.
```
# snmpget -v 2c -c public pdu1.gc134.ops.releng.mdc2.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.5.1.1.3
iso.3.6.1.4.1.1718.3.2.3.1.5.1.1.3 = INTEGER: 1
# snmpset -v 2c -c secret pdu1.gc134.ops.releng.mdc2.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.11.1.1.3 i 2
iso.3.6.1.4.1.1718.3.2.3.1.11.1.1.3 = INTEGER: 2
# snmpget -v 2c -c public pdu1.gc134.ops.releng.mdc2.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.5.1.1.3              iso.3.6.1.4.1.1718.3.2.3.1.5.1.1.3 = INTEGER: 0
# snmpget -v 2c -c public pdu1.gc134.ops.releng.mdc2.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.5.1.1.3
iso.3.6.1.4.1.1718.3.2.3.1.5.1.1.3 = INTEGER: 0
# snmpset -v 2c -c secret pdu1.gc134.ops.releng.mdc2.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.11.1.1.3 i 1
iso.3.6.1.4.1.1718.3.2.3.1.11.1.1.3 = INTEGER: 1
# snmpget -v 2c -c public pdu1.gc134.ops.releng.mdc2.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.5.1.1.3              iso.3.6.1.4.1.1718.3.2.3.1.5.1.1.3 = INTEGER: 1
```

```
[[e-mail] relops-infra]$ ping -c1 t-yosemite-r7-451.test.releng.mdc1.mozilla.com; ssh -vv [e-mail]
PING t-yosemite-r7-451.test.releng.mdc1.mozilla.com (10.49.56.75) 56(84) bytes of data.

--- t-yosemite-r7-451.test.releng.mdc1.mozilla.com ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 10000ms

OpenSSH_6.6, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /home/dhouse/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 4: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to t-yosemite-r7-451.test.releng.mdc1.mozilla.com [10.49.56.75] port 22.
debug1: connect to address 10.49.56.75 port 22: Connection timed out
ssh: connect to host t-yosemite-r7-451.test.releng.mdc1.mozilla.com port 22: Connection timed out
```

Comment 3

[:dhouse]

Opened a case with the datacenter remote-hands to physically restart the machine and look for the login screen (and kick off reimage if boot fails): RITM0264839

Comment 4

[:dhouse]

completed and the machine looks good