1480658 - (t-yosemite-r7-045) [MDC2] t-yosemite-r7-045 Problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Adrian Pop]

Today, when I wanted to log on the machine t-yosemite-r7-045, I've received the following error :

Stdio forwarding request failed: Session open refused by peer
ssh_exchange_identification: Connection closed by remote host

Comment 1

[Bogdan Crisan [:bcrisan] (EEST - GMT + 3)]

Problem persists, also the machine doesn't appear in taskcluster.

Comment 2

[:dhouse]

Rebooted and it is taking tasks:
https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-osx-1010/workers/mdc2/t-yosemite-r7-045

Comment 3

[Danut Labici [:dlabici]]

Hey Van,

Seems like the machine has the STDIO issue again. Can you please restart it (or someone in dcops. Dave is in PTO and not accepting NI?s 
Tried to SSH into it - failed
Tried to use roller to restart - failed.

Comment 4

[Van Le [:van]]

opened REQ0246655 with QTS for reimage.

Comment 5

[Van Le [:van]]

back online.

vle@DESKTOP-3HK51T3:~$ fping t-yosemite-r7-045.test.releng.mdc2.mozilla.com
t-yosemite-r7-045.test.releng.mdc2.mozilla.com is alive

Comment 6

[Roland Mutter Michael (:rmutter)]

Reopening this as it looks like the server is offline. It won't respond to pings.

Comment 7

[Roland Mutter Michael (:rmutter)]

Looks good now , responds to ping and can ssh into it.

Comment 8

[Roland Mutter Michael (:rmutter)]

Seems like the host is offline once again. Won't respond to pings.

Comment 9

[Radu Iman[:riman]]

The machine looks good now:
https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-osx-1010/workers/mdc2/t-yosemite-r7-045

Comment 10

[Roland Mutter Michael (:rmutter)]

Seems like we are facing STDIO problem on this machine once again. Reopening.

Comment 11

[Adrian Pop]

It seems that the machine still has STDIO issue. Can you please check again ?

Comment 12

[Van Le [:van]]

opened REQ0252727 with QTS for reimage.

Comment 13

[Danut Labici [:dlabici]]

Machine is back and running jobs:
https://tools.taskcluster.net/groups/CUMQ99uhRYKZh_X5yGpFRQ/tasks/f_RhIiVYS4KJHoERKCaHmA/runs/0

Comment 14

[Roland Mutter Michael (:rmutter)]

Seems like the machine is in a bad state once again. Last job ended in exception ,we're receiving stdio issue when trying to ssh and machine doesn't respond to ping.

Comment 15

[Radu Iman[:riman]]

Added this machine back in TC using the quarantine_tc.py script [1].  I've rebooted it by roller.

It looks good now:
https://tools.taskcluster.net/groups/c28OglDCQQSqMgIZg3jr7g/tasks/bNTxPfuXQbuXklJ95sF-jA/runs/0

[1] - https://wiki.mozilla.org/Connect_and_Troubleshoot_workers_in_CI#How_to_add.2Fdefine_a_worker_if_it_is_missing_from_Taskcluster

Comment 16

[Roland Mutter Michael (:rmutter)]

Reopening this as it seems that we're hitting STDIO issue and the worker is not pingable.

Comment 17

[Roland Mutter Michael (:rmutter)]

Reopening this as it seems like we're hitting STDIO issue and workers seems unreachable.

Comment 18

[:dhouse]

https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-osx-1010/workers/mdc2/t-yosemite-r7-045

fixed. rebooted

Comment 19

[Roland Mutter Michael (:rmutter)]

Reopening this as it seems like we're hitting STDIO issue and worker seems unreachable.

Comment 20

[Adrian Pop]

the machine seems to be up and running and taking jobs.
https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-osx-1010/workers/mdc2/t-yosemite-r7-045
We will close the bug for now. If the problem will persist in the future, we will re-open this bug.