711374 - diagnose then reboot and maybe reimage talos-r4-snow-018

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[John Ford [:jhford] CET/CEST Berlin Time]

This machine is taking a *LONG* time to auth with ssh and screensharing is not working.  I verified with Instruments System trace that the ssh client process is doing nothing at all.

Before touching the machine, please check the power LED to and let me know the status.  Please also connect a display and let me know what state the machine is in.  Once that's done, if there are no further diagnostics possible, please reconnect the dongle then reboot.  If the machine is not reachable by SSH and VNC, please reimage this machine.

~ $ ping -c 10 talos-r4-snow-018.build.mozilla.org
PING talos-r4-snow-018.build.scl1.mozilla.com (10.12.51.46): 56 data bytes
64 bytes from 10.12.51.46: icmp_seq=0 ttl=60 time=87.378 ms
64 bytes from 10.12.51.46: icmp_seq=1 ttl=60 time=250.019 ms
64 bytes from 10.12.51.46: icmp_seq=2 ttl=60 time=212.425 ms
64 bytes from 10.12.51.46: icmp_seq=3 ttl=60 time=173.251 ms
64 bytes from 10.12.51.46: icmp_seq=4 ttl=60 time=90.108 ms
64 bytes from 10.12.51.46: icmp_seq=5 ttl=60 time=89.449 ms
64 bytes from 10.12.51.46: icmp_seq=6 ttl=60 time=89.466 ms
64 bytes from 10.12.51.46: icmp_seq=7 ttl=60 time=89.377 ms
64 bytes from 10.12.51.46: icmp_seq=8 ttl=60 time=89.827 ms
64 bytes from 10.12.51.46: icmp_seq=9 ttl=60 time=86.555 ms

--- talos-r4-snow-018.build.scl1.mozilla.com ping statistics ---
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 86.555/125.785/250.019/58.939 ms

~ $ ssh -v talos-r4-snow-018.build.mozilla.org
OpenSSH_5.6p1, OpenSSL 0.9.8r 8 Feb 2011
debug1: Reading configuration data /Users/jhford/.ssh/config
debug1: Applying options for *.build.mozilla.org
debug1: Applying options for *mozilla.org
debug1: Reading configuration data /etc/ssh_config
debug1: Applying options for *
debug1: Connecting to talos-r4-snow-018.build.mozilla.org [10.12.51.46] port 22.
debug1: Connection established.
debug1: identity file /Users/jhford/.ssh/jhford-mozilla type 2
debug1: identity file /Users/jhford/.ssh/jhford-mozilla-cert type -1
debug1: identity file /Users/jhford/.ssh/jhford-mozilla type 2
debug1: identity file /Users/jhford/.ssh/jhford-mozilla-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.2
debug1: match: OpenSSH_5.2 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.6
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host 'talos-r4-snow-018.build.mozilla.org' is known and matches the RSA host key.
debug1: Found key in /Users/jhford/.ssh/known_hosts:733
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering DSA public key: /Users/jhford/.ssh/jhford-mozilla
debug1: Server accepts key: pkalg ssh-dss blen 433

Comment 1

[Jake Watkins [:dividehex]]

LED was on and solid.  Display state showed no vga signal and pressing keys on the keyboard did not wake it up. I reconnect the DVI Doctor and rebooted.  SSH and VNC are accessible and don't show any signs of sluggishness.