1474268 - (t-yosemite-r7-124) [MDC2] t-yosemite-r7-124 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Comment 1

[Danut Labici [:dlabici]]

Machine had issues again, so I have re-imaged it.

:zfay can you check if the machine took jobs?
Thanks.

Comment 2

[Zsolt Fay [:zfay]]

The machine is not in TC anymore. Any attempt to SSH into it results in:

Stdio forwarding request failed: Session open refused by peer
ssh_exchange_identification: Connection closed by remote host

Comment 3

[Bogdan Crisan [:bcrisan] (EEST - GMT + 3)]

(In reply to Zsolt Fay [:zsoltfay] from comment #2)
> The machine is not in TC anymore. Any attempt to SSH into it results in:
> 
> Stdio forwarding request failed: Session open refused by peer
> ssh_exchange_identification: Connection closed by remote host
Same here^^


bcrisan@bcrisan-P6198:~$ fping t-yosemite-r7-124.test.releng.mdc2.mozilla.com
t-yosemite-r7-124.test.releng.mdc2.mozilla.com is unreachable

Comment 4

[:dhouse]

No response to ping or ssh. I cycled the power on it, and it did not start responding to ping after that either.

https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-osx-1010/workers/mdc2/t-yosemite-r7-124
https://papertrailapp.com/groups/1223184/events?q=t-yosemite-r7-124&focus=957969798439002156

THe last deploystudio log I found is from May 24th (May 24, 8:54 AM):
```
The workflow 'Restore yosemite-r7' was launched on the computer C07Q73L4G1J2 (name: t-yosemite-r7-124.test.releng.mdc2.mozilla.com, ip: 10.51.56.148, mac: 38:c9:86:04:be:ac) with a SUCCESSFUL termination status. This mail was generated automatically by DeployStudio Server. -- The DeployStudio Team.
```

With a puppet cert Thu, May 24, 9:09 AM:
```
Creating certificate for t-yosemite-r7-129.test.releng.mdc2.mozilla.com (10.51.56.153) from 10.51.56.153.
```

It looks like the machine was running fine after that last reimage because there is a puppet email; I did not check papertrail archives for when it stopped working.
```
Puppet Report for t-yosemite-r7-124.test.releng.mdc2.mozilla.com
Thu, Jun 28, 12:10 PM
to "[e-mail]"
Thu Jun 28 11:10:02 -0700 2018 Puppet (err): Could not retrieve catalog from remote server: end of file reached
Thu Jun 28 11:10:02 -0700 2018 Puppet (err): Could not retrieve catalog; skipping run
```

Comment 5

[:dhouse]

I've created a bug for DCOps to physically check the machine and reimage it.

(All systems) https://papertrailapp.com/groups/1141234/events?q=t-yosemite-r7-124&focus=957969798439002156
Machine has not logged in papertrail's 2-week window: https://papertrailapp.com/groups/1223184?filter=t-yosemite-r7-124

Comment 6

[:dhouse]

This machine has completed 3 tasks since being reimaged by QTS, and it is currently running the worker on a task and reporting logs to papertrail.
https://papertrailapp.com/systems/2118878102/events
https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-osx-1010/workers/mdc2/t-yosemite-r7-124

Comment 7

[Bogdan Crisan [:bcrisan] (EEST - GMT + 3)]

The worker is alive but no response from ssh or roller reboot.

Comment 8

[:dhouse]

looks good right now
https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-osx-1010/workers/mdc2/t-yosemite-r7-124