Closed Bug 1548457 Opened 7 years ago Closed 7 years ago

[MDC2] t-yosemite-r7-024 is unreachable

Categories

(Infrastructure & Operations :: RelOps: Hardware, defect)

Product:
Infrastructure & Operations
Component:
RelOps: Hardware
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: relops-bug-generator, Unassigned)

References

Details

Reboot t-yosemite-r7-024.test.releng.mdc2.mozilla.com. 10.51.56.183
Requested by mozilla-auth0/ad|Mozilla-LDAP|apop
Relops controller action failed:
2019-05-02T05:53:17.996893 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-05-02T05:56:32.358054 snmp_reboot pdu1.gc134.ops.releng.mdc2.mozilla.com bc15 --delay 60 Exception
2019-05-02T05:56:32.364040 ipmi ipmi_reset KeyError
2019-05-02T05:56:32.367481 ipmi ipmi_cycle KeyError
2019-05-02T05:56:32.370563 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-024.test.releng.mdc2.mozilla.com. 10.51.56.183
Requested by mozilla-auth0/ad|Mozilla-LDAP|apop
Relops controller action failed:
2019-05-02T10:45:06.657121 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-05-02T10:48:20.995922 snmp_reboot pdu1.gc134.ops.releng.mdc2.mozilla.com bc15 --delay 60 Exception
2019-05-02T10:48:20.999429 ipmi ipmi_reset KeyError
2019-05-02T10:48:21.001582 ipmi ipmi_cycle KeyError
2019-05-02T10:48:21.003204 ipmi ipmi_on KeyError

Tried to connect on it, but I've been welcomed with the following message :

Stdio forwarding request failed: Session open refused by peer
ssh_exchange_identification: Connection closed by remote host

I've checked the papertrail and the last entries were :

May 01 14:19:31 t-yosemite-r7-024 kernel: firefox (map: 0xffffff803ce16780) triggered DYLD shared region unnest for map: 0xffffff803ce16780, region 0x7fff82a00000->0x7fff82c00000. While not abnormal for debuggers, this increases system memory footprint until the target exits.
May 01 14:19:53 t-yosemite-r7-024 kernel: firefox (map: 0xffffff803ce16780) triggered DYLD shared region unnest for map: 0xffffff803ce16780, region 0x7fff82a00000->0x7fff82c00000. While not abnormal for debuggers, this increases system memory footprint until the target exits.
May 01 14:20:13 t-yosemite-r7-024 kernel: firefox (map: 0xffffff803ce16780) triggered DYLD shared region unnest for map: 0xffffff803ce16780, region 0x7fff82a00000->0x7fff82c00000. While not abnormal for debuggers, this increases system memory footprint until the target exits.
May 01 14:20:29 t-yosemite-r7-024 kernel: firefox (map: 0xffffff803ce16780) triggered DYLD shared region unnest for map: 0xffffff803ce16780, region 0x7fff82a00000->0x7fff82c00000. While not abnormal for debuggers, this increases system memory footprint until the target exits.
May 01 14:20:46 t-yosemite-r7-024 kernel: firefox (map: 0xffffff803ce16780) triggered DYLD shared region unnest for map: 0xffffff803ce16780, region 0x7fff82a00000->0x7fff82c00000. While not abnormal for debuggers, this increases system memory footprint until the target exits.
May 01 14:21:00 t-yosemite-r7-024 kernel: firefox (map: 0xffffff803ce16780) triggered DYLD shared region unnest for map: 0xffffff803ce16780, region 0x7fff82a00000->0x7fff82c00000. While not abnormal for debuggers, this increases system memory footprint until the target exits.
May 01 14:21:19 t-yosemite-r7-024 kernel: firefox (map: 0xffffff803ce16780) triggered DYLD shared region unnest for map: 0xffffff803ce16780, region 0x7fff82a00000->0x7fff82c00000. While not abnormal for debuggers, this increases system memory footprint until the target exits.
May 01 14:22:37 t-yosemite-r7-024 kernel: firefox (map: 0xffffff803ce16780) triggered DYLD shared region unnest for map: 0xffffff803ce16780, region 0x7fff82a00000->0x7fff82c00000. While not abnormal for debuggers, this increases system memory footprint until the target exits.
May 01 14:23:04 t-yosemite-r7-024 kernel: firefox (map: 0xffffff803ce16780) triggered DYLD shared region unnest for map: 0xffffff803ce16780, region 0x7fff82a00000->0x7fff82c00000. While not abnormal for debuggers, this increases system memory footprint until the target exits.
May 01 14:23:35 t-yosemite-r7-024 kernel: firefox (map: 0xffffff803ce16780) triggered DYLD shared region unnest for map: 0xffffff803ce16780, region 0x7fff82a00000->0x7fff82c00000. While not abnormal for debuggers, this increases system memory footprint until the target exits.
May 01 14:23:51 t-yosemite-r7-024 kernel: firefox (map: 0xffffff803ce16780) triggered DYLD shared region unnest for map: 0xffffff803ce16780, region 0x7fff82a00000->0x7fff82c00000. While not abnormal for debuggers, this increases system memory footprint until the target exits.
May 01 14:24:23 t-yosemite-r7-024 kernel: firefox (map: 0xffffff803ce16780) triggered DYLD shared region unnest for map: 0xffffff803ce16780, region 0x7fff82a00000->0x7fff82c00000. While not abnormal for debuggers, this increases system memory footprint until the target exits.
May 01 14:24:56 t-yosemite-r7-024 kernel: firefox (map: 0xffffff803ce16780) triggered DYLD shared region unnest for map: 0xffffff803ce16780, region 0x7fff82a00000->0x7fff82c00000. While not abnormal for debuggers, this increases system memory footprint until the target exits.
May 01 14:25:47 t-yosemite-r7-024 kernel: firefox (map: 0xffffff803ce16780) triggered DYLD shared region unnest for map: 0xffffff803ce16780, region 0x7fff82a00000->0x7fff82c00000. While not abnormal for debuggers, this increases system memory footprint until the target exits.
May 01 14:26:33 t-yosemite-r7-024 kernel: Kext loading now disabled.
May 01 14:26:33 t-yosemite-r7-024 kernel: Kext unloading now disabled.
May 01 14:26:33 t-yosemite-r7-024 kernel: Kext autounloading now disabled.
May 01 14:26:33 t-yosemite-r7-024 kernel: Kernel requests now disabled.

Dave, can you please take a look?

Flags: needinfo?(dhouse)

Reboot t-yosemite-r7-024.test.releng.mdc2.mozilla.com. 10.51.56.183
Requested by mozilla-auth0/ad|Mozilla-LDAP|dhouse
Relops controller action failed:
2019-05-02T15:26:10.247729 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-05-02T15:29:24.588149 snmp_reboot pdu1.gc134.ops.releng.mdc2.mozilla.com bc15 --delay 60 Exception
2019-05-02T15:29:24.594347 ipmi ipmi_reset KeyError
2019-05-02T15:29:24.597912 ipmi ipmi_cycle KeyError
2019-05-02T15:29:24.600815 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-024.test.releng.mdc2.mozilla.com. 10.51.56.183
Requested by mozilla-auth0/ad|Mozilla-LDAP|dlabici
Relops controller action failed:
2019-05-03T07:32:46.238169 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-05-03T07:36:00.566092 snmp_reboot pdu1.gc134.ops.releng.mdc2.mozilla.com bc15 --delay 60 Exception
2019-05-03T07:36:00.572354 ipmi ipmi_reset KeyError
2019-05-03T07:36:00.576297 ipmi ipmi_cycle KeyError
2019-05-03T07:36:00.578996 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-024.test.releng.mdc2.mozilla.com. 10.51.56.183
Requested by mozilla-auth0/ad|Mozilla-LDAP|riman
Relops controller action failed:
2019-05-03T19:18:37.020660 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-05-03T19:21:51.428107 snmp_reboot pdu1.gc134.ops.releng.mdc2.mozilla.com bc15 --delay 60 Exception
2019-05-03T19:21:51.431962 ipmi ipmi_reset KeyError
2019-05-03T19:21:51.435788 ipmi ipmi_cycle KeyError
2019-05-03T19:21:51.438854 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-024.test.releng.mdc2.mozilla.com. 10.51.56.183
Requested by mozilla-auth0/ad|Mozilla-LDAP|acraciun
Relops controller action failed:
2019-05-06T08:56:28.279322 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-05-06T08:59:42.667258 snmp_reboot pdu1.gc134.ops.releng.mdc2.mozilla.com bc15 --delay 60 Exception
2019-05-06T08:59:42.670583 ipmi ipmi_reset KeyError
2019-05-06T08:59:42.672617 ipmi ipmi_cycle KeyError
2019-05-06T08:59:42.674380 ipmi ipmi_on KeyError

Need a power cycle from PDU.

Reboot t-yosemite-r7-024.test.releng.mdc2.mozilla.com. 10.51.56.183
Requested by mozilla-auth0/ad|Mozilla-LDAP|acraciun
Relops controller action failed:
2019-05-07T13:53:12.671319 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-05-07T13:56:26.999022 snmp_reboot pdu1.gc134.ops.releng.mdc2.mozilla.com bc15 --delay 60 Exception
2019-05-07T13:56:27.001280 ipmi ipmi_reset KeyError
2019-05-07T13:56:27.002748 ipmi ipmi_cycle KeyError
2019-05-07T13:56:27.004240 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-024.test.releng.mdc2.mozilla.com. 10.51.56.183
Requested by mozilla-auth0/ad|Mozilla-LDAP|acraciun
Relops controller action failed:
2019-05-09T14:06:16.727540 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-05-09T14:09:31.067521 snmp_reboot pdu1.gc134.ops.releng.mdc2.mozilla.com bc15 --delay 60 Exception
2019-05-09T14:09:31.070738 ipmi ipmi_reset KeyError
2019-05-09T14:09:31.072346 ipmi ipmi_cycle KeyError
2019-05-09T14:09:31.074025 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-024.test.releng.mdc2.mozilla.com. 10.51.56.183
Requested by mozilla-auth0/ad|Mozilla-LDAP|apop
Relops controller action failed:
2019-05-12T00:58:45.622089 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-05-12T01:01:59.980057 snmp_reboot pdu1.gc134.ops.releng.mdc2.mozilla.com bc15 --delay 60 Exception
2019-05-12T01:01:59.985865 ipmi ipmi_reset KeyError
2019-05-12T01:01:59.990525 ipmi ipmi_cycle KeyError
2019-05-12T01:01:59.992133 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-024.test.releng.mdc2.mozilla.com. 10.51.56.183
Requested by mozilla-auth0/ad|Mozilla-LDAP|acraciun
Relops controller action failed:
2019-05-13T12:03:43.709135 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-05-13T12:06:58.023737 snmp_reboot pdu1.gc134.ops.releng.mdc2.mozilla.com bc15 --delay 60 Exception
2019-05-13T12:06:58.025824 ipmi ipmi_reset KeyError
2019-05-13T12:06:58.027280 ipmi ipmi_cycle KeyError
2019-05-13T12:06:58.028630 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-024.test.releng.mdc2.mozilla.com. 10.51.56.183
Requested by mozilla-auth0/ad|Mozilla-LDAP|apop
Relops controller action failed:
2019-05-16T02:18:31.925243 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-05-16T02:21:46.248458 snmp_reboot pdu1.gc134.ops.releng.mdc2.mozilla.com bc15 --delay 60 Exception
2019-05-16T02:21:46.251417 ipmi ipmi_reset KeyError
2019-05-16T02:21:46.252918 ipmi ipmi_cycle KeyError
2019-05-16T02:21:46.254578 ipmi ipmi_on KeyError

the machine seems to be up and running and taking jobs.
https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-osx-1010/workers/mdc2/t-yosemite-r7-024
We will close the bug for now. If the problem will persist in the future, we will re-open this bug.

Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED

worker is taking jobs

Flags: needinfo?(dhouse)
You need to log in before you can comment on or make changes to this bug.