Closed Bug 1542116 Opened 7 years ago Closed 6 years ago

[MDC1] t-yosemite-r7-300 is unreachable

Categories

(Infrastructure & Operations :: RelOps: Hardware, defect)

Product:
Infrastructure & Operations
Component:
RelOps: Hardware
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: relops-bug-generator, Assigned: dividehex)

References

Details

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|acraciun
Relops controller action failed:
2019-04-05T06:20:42.780385 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-04-05T06:23:56.934776 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-04-05T06:23:56.937714 ipmi ipmi_reset KeyError
2019-04-05T06:23:56.939098 ipmi ipmi_cycle KeyError
2019-04-05T06:23:56.940682 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|acraciun
Relops controller action failed:
2019-04-05T13:39:08.909900 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-04-05T13:42:23.061930 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-04-05T13:42:23.064635 ipmi ipmi_reset KeyError
2019-04-05T13:42:23.066629 ipmi ipmi_cycle KeyError
2019-04-05T13:42:23.067966 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|acraciun
Relops controller action failed:
2019-04-08T12:10:52.850376 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-04-08T12:14:07.037628 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-04-08T12:14:07.040435 ipmi ipmi_reset KeyError
2019-04-08T12:14:07.042165 ipmi ipmi_cycle KeyError
2019-04-08T12:14:07.043939 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|acraciun
Relops controller action failed:
2019-04-09T06:58:54.322637 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-04-09T07:02:08.449923 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-04-09T07:02:08.453294 ipmi ipmi_reset KeyError
2019-04-09T07:02:08.456023 ipmi ipmi_cycle KeyError
2019-04-09T07:02:08.462081 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|acraciun
Relops controller action failed:
2019-04-12T08:00:27.503032 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-04-12T08:03:41.670659 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-04-12T08:03:41.675799 ipmi ipmi_reset KeyError
2019-04-12T08:03:41.678242 ipmi ipmi_cycle KeyError
2019-04-12T08:03:41.680561 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|acraciun
Relops controller action failed:
2019-04-15T12:21:50.421906 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-04-15T12:25:04.608332 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-04-15T12:25:04.613187 ipmi ipmi_reset KeyError
2019-04-15T12:25:04.615855 ipmi ipmi_cycle KeyError
2019-04-15T12:25:04.618483 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|acraciun
Relops controller action failed:
2019-04-16T09:01:42.219353 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-04-16T09:04:56.391105 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-04-16T09:04:56.394059 ipmi ipmi_reset KeyError
2019-04-16T09:04:56.395783 ipmi ipmi_cycle KeyError
2019-04-16T09:04:56.397416 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|acraciun
Relops controller action failed:
2019-04-17T12:45:24.014401 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-04-17T12:48:38.172376 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-04-17T12:48:38.174558 ipmi ipmi_reset KeyError
2019-04-17T12:48:38.176109 ipmi ipmi_cycle KeyError
2019-04-17T12:48:38.177479 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|apop
Relops controller action failed:
2019-04-18T06:18:45.464802 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-04-18T06:21:59.636516 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-04-18T06:21:59.639145 ipmi ipmi_reset KeyError
2019-04-18T06:21:59.640686 ipmi ipmi_cycle KeyError
2019-04-18T06:21:59.642180 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|apop
Relops controller action failed:
2019-04-18T07:59:51.110032 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-04-18T08:03:05.276060 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-04-18T08:03:05.281088 ipmi ipmi_reset KeyError
2019-04-18T08:03:05.282578 ipmi ipmi_cycle KeyError
2019-04-18T08:03:05.284072 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|acraciun
Relops controller action failed:
2019-04-19T06:06:51.274070 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-04-19T06:10:05.484717 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-04-19T06:10:05.490909 ipmi ipmi_reset KeyError
2019-04-19T06:10:05.494369 ipmi ipmi_cycle KeyError
2019-04-19T06:10:05.498676 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|acraciun
Relops controller action failed:
2019-04-22T13:07:39.177243 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-04-22T13:10:53.372988 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-04-22T13:10:53.377041 ipmi ipmi_reset KeyError
2019-04-22T13:10:53.379581 ipmi ipmi_cycle KeyError
2019-04-22T13:10:53.382086 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|acraciun
Relops controller action failed:
2019-04-24T08:22:33.390192 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-04-24T08:25:47.507547 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-04-24T08:25:47.512852 ipmi ipmi_reset KeyError
2019-04-24T08:25:47.515286 ipmi ipmi_cycle KeyError
2019-04-24T08:25:47.517522 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|acraciun
Relops controller action failed:
2019-04-25T13:28:46.617701 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-04-25T13:32:00.763874 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-04-25T13:32:00.769594 ipmi ipmi_reset KeyError
2019-04-25T13:32:00.773050 ipmi ipmi_cycle KeyError
2019-04-25T13:32:00.776480 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|dlabici
Relops controller action failed:
2019-04-29T04:16:56.359490 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-04-29T04:20:10.584712 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-04-29T04:20:10.591136 ipmi ipmi_reset KeyError
2019-04-29T04:20:10.594571 ipmi ipmi_cycle KeyError
2019-04-29T04:20:10.598031 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|bcrisan
Relops controller action failed:
2019-05-03T07:39:19.325794 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-05-03T07:42:33.494422 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-05-03T07:42:33.498970 ipmi ipmi_reset KeyError
2019-05-03T07:42:33.500277 ipmi ipmi_cycle KeyError
2019-05-03T07:42:33.501597 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|acraciun
Relops controller action failed:
2019-05-07T13:49:17.531821 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-05-07T13:52:31.673096 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-05-07T13:52:31.682100 ipmi ipmi_reset KeyError
2019-05-07T13:52:31.687264 ipmi ipmi_cycle KeyError
2019-05-07T13:52:31.689746 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|acraciun
Relops controller action failed:
2019-05-07T14:03:32.598410 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-05-07T14:06:46.802384 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-05-07T14:06:46.807644 ipmi ipmi_reset KeyError
2019-05-07T14:06:46.809144 ipmi ipmi_cycle KeyError
2019-05-07T14:06:46.810634 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|acraciun
Relops controller action failed:
2019-05-09T14:01:11.089393 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-05-09T14:04:25.268860 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-05-09T14:04:25.274566 ipmi ipmi_reset KeyError
2019-05-09T14:04:25.276486 ipmi ipmi_cycle KeyError
2019-05-09T14:04:25.278354 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|acraciun
Relops controller action failed:
2019-05-13T11:59:57.281088 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-05-13T12:03:11.434806 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-05-13T12:03:11.438034 ipmi ipmi_reset KeyError
2019-05-13T12:03:11.439566 ipmi ipmi_cycle KeyError
2019-05-13T12:03:11.441053 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|apop
Relops controller action failed:
2019-05-16T02:24:03.732987 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-05-16T02:27:18.946018 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-05-16T02:27:18.951690 ipmi ipmi_reset KeyError
2019-05-16T02:27:18.953194 ipmi ipmi_cycle KeyError
2019-05-16T02:27:18.954855 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|apop
Relops controller action failed:
2019-05-20T15:29:40.908322 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-05-20T15:32:55.034306 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-05-20T15:32:55.036683 ipmi ipmi_reset KeyError
2019-05-20T15:32:55.039368 ipmi ipmi_cycle KeyError
2019-05-20T15:32:55.042072 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|acraciun
Relops controller action failed:
2019-05-24T11:11:11.636990 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-05-24T11:14:25.827393 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-05-24T11:14:25.830202 ipmi ipmi_reset KeyError
2019-05-24T11:14:25.831843 ipmi ipmi_cycle KeyError
2019-05-24T11:14:25.833371 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|apop
Relops controller action failed:
2019-05-28T11:38:53.634882 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-05-28T11:42:07.844773 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-05-28T11:42:07.850482 ipmi ipmi_reset KeyError
2019-05-28T11:42:07.852809 ipmi ipmi_cycle KeyError
2019-05-28T11:42:07.854444 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|acraciun
Relops controller action failed:
2019-05-30T07:19:39.234108 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-05-30T07:22:53.388843 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-05-30T07:22:53.393988 ipmi ipmi_reset KeyError
2019-05-30T07:22:53.397388 ipmi ipmi_cycle KeyError
2019-05-30T07:22:53.400805 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|acraciun
Relops controller action failed:
2019-06-06T14:29:58.953949 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-06-06T14:33:13.122122 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-06-06T14:33:13.124669 ipmi ipmi_reset KeyError
2019-06-06T14:33:13.126992 ipmi ipmi_cycle KeyError
2019-06-06T14:33:13.129197 ipmi ipmi_on KeyError

Assignee: nobody → jwatkins
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.