Closed Bug 1539109 Opened 7 years ago Closed 7 years ago

[MDC1] t-yosemite-r7-300 is unreachable

Categories

(Infrastructure & Operations :: RelOps: Hardware, task)

Product:
Infrastructure & Operations
Component:
RelOps: Hardware
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: relops-bug-generator, Unassigned)

References

Details

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|riman
Relops controller action failed:
2019-03-26T11:02:19.365702 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-03-26T11:05:33.553326 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-03-26T11:05:33.560481 ipmi ipmi_reset KeyError
2019-03-26T11:05:33.563915 ipmi ipmi_cycle KeyError
2019-03-26T11:05:33.565474 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|apop
Relops controller action failed:
2019-03-26T19:06:05.623667 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-03-26T19:09:19.778158 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-03-26T19:09:19.780432 ipmi ipmi_reset KeyError
2019-03-26T19:09:19.781848 ipmi ipmi_cycle KeyError
2019-03-26T19:09:19.783357 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|apop
Relops controller action failed:
2019-03-26T23:37:57.379110 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-03-26T23:41:11.534234 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-03-26T23:41:11.538060 ipmi ipmi_reset KeyError
2019-03-26T23:41:11.540390 ipmi ipmi_cycle KeyError
2019-03-26T23:41:11.541925 ipmi ipmi_on KeyError

rebooted via roller, but it didn't helped
tried to log on it via ssh but received :

Stdio forwarding request failed: Session open refused by peer
ssh_exchange_identification: Connection closed by remote host

tried to ping the machine :

Pinging t-yosemite-r7-300.test.releng.mdc1.mozilla.com [10.49.56.241] with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 10.49.56.241:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

the last entry on papertrail :

Mar 25 16:09:11 t-yosemite-r7-300.test.releng.mdc1.mozilla.com taskgated: binary have embedded signature that validated /Users/cltbld/tasks/task_1553555232/build/venv/bin/python[753]
Mar 25 16:09:12 t-yosemite-r7-300.test.releng.mdc1.mozilla.com taskgated: binary have embedded signature that validated /Users/cltbld/tasks/task_1553555232/build/venv/bin/python[757]
Mar 25 16:09:24 t-yosemite-r7-300.test.releng.mdc1.mozilla.com taskgated: binary have embedded signature that validated /System/Library/CoreServices/backupd.bundle/Contents/Resources/TMHelperAgent.app[776]
Mar 25 16:09:24 t-yosemite-r7-300.test.releng.mdc1.mozilla.com TMHelperAgent: LSExceptions [0x7fe260f15620] loaded
Mar 25 16:09:34 t-yosemite-r7-300.test.releng.mdc1.mozilla.com TMHelperAgent: LSExceptions [0x7fe260f15620] unloaded
Mar 25 16:09:38 t-yosemite-r7-300.test.releng.mdc1.mozilla.com taskgated: binary have embedded signature that validated /Users/cltbld/tasks/task_1553555232/build/venv/bin/python[783]
Mar 25 16:09:38 t-yosemite-r7-300.test.releng.mdc1.mozilla.com Dock: LSExceptions [0x6000002b4ac0] loaded
Mar 25 16:09:39 t-yosemite-r7-300.test.releng.mdc1.mozilla.com firefox: LSExceptions [0x1098809a0] loaded
Mar 25 16:09:43 t-yosemite-r7-300.test.releng.mdc1.mozilla.com sandboxd: ([791]): plugin-container(791) deny forbidden-sandbox-reinit
Mar 25 16:09:44 t-yosemite-r7-300.test.releng.mdc1.mozilla.com sandboxd: ([792]): plugin-container(792) deny forbidden-sandbox-reinit
Mar 25 16:09:46 t-yosemite-r7-300.test.releng.mdc1.mozilla.com sandboxd: ([793]): plugin-container(793) deny forbidden-sandbox-reinit
Mar 25 16:09:49 t-yosemite-r7-300.test.releng.mdc1.mozilla.com Dock: LSExceptions [0x6000002b4ac0] unloaded
Mar 25 16:09:49 t-yosemite-r7-300.test.releng.mdc1.mozilla.com firefox: LSExceptions [0x1098809a0] unloaded
Mar 25 16:09:50 t-yosemite-r7-300.test.releng.mdc1.mozilla.com Install: in Progress[795]: LSExceptions [0x7f8158e014e0] loaded
Mar 25 16:09:50 t-yosemite-r7-300.test.releng.mdc1.mozilla.com diagnostics_agent: message repeated 13 times: [ ]
Mar 25 16:09:50 t-yosemite-r7-300.test.releng.mdc1.mozilla.com diagnostics_agent: message repeated 3 times: [ ]
Mar 25 16:09:50 t-yosemite-r7-300.test.releng.mdc1.mozilla.com spindump: [793] Not monitoring spin due to thermal pressure
Mar 25 16:09:50 t-yosemite-r7-300.test.releng.mdc1.mozilla.com taskgated: binary have embedded signature that validated /System/Library/CoreServices/SubmitDiagInfo[800]
Mar 25 16:09:50 t-yosemite-r7-300.test.releng.mdc1.mozilla.com SubmitDiagInfo: Couldn't load config file from on-disk location. Falling back to default location. Reason: Won't serialize in _readDictionaryFromJSONData due to nil object
Mar 25 16:09:51 t-yosemite-r7-300.test.releng.mdc1.mozilla.com spindump: [791] Not monitoring spin due to thermal pressure
Mar 25 16:09:51 t-yosemite-r7-300.test.releng.mdc1.mozilla.com spindump: [787] Not monitoring spin due to thermal pressure

Dave, can you please take a look ?

Flags: needinfo?(dhouse)

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|dlabici
Relops controller action failed:
2019-03-28T00:26:23.371212 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-03-28T00:29:37.589548 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-03-28T00:29:37.595069 ipmi ipmi_reset KeyError
2019-03-28T00:29:37.598516 ipmi ipmi_cycle KeyError
2019-03-28T00:29:37.601513 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|acraciun
Relops controller action failed:
2019-03-28T07:05:34.188617 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-03-28T07:08:48.393541 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-03-28T07:08:48.399430 ipmi ipmi_reset KeyError
2019-03-28T07:08:48.402016 ipmi ipmi_cycle KeyError
2019-03-28T07:08:48.405063 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|apop
Relops controller action failed:
2019-03-28T15:09:05.706523 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-03-28T15:12:19.896552 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-03-28T15:12:19.902675 ipmi ipmi_reset KeyError
2019-03-28T15:12:19.906319 ipmi ipmi_cycle KeyError
2019-03-28T15:12:19.909925 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|acraciun
Relops controller action failed:
2019-03-29T07:17:26.408830 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-03-29T07:20:40.625464 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-03-29T07:20:40.631240 ipmi ipmi_reset KeyError
2019-03-29T07:20:40.633354 ipmi ipmi_cycle KeyError
2019-03-29T07:20:40.635398 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|acraciun
Relops controller action failed:
2019-03-29T11:33:28.565602 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-03-29T11:36:42.728238 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-03-29T11:36:42.731081 ipmi ipmi_reset KeyError
2019-03-29T11:36:42.732589 ipmi ipmi_cycle KeyError
2019-03-29T11:36:42.734104 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|zfay
Relops controller action failed:
2019-03-30T10:20:23.895391 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-03-30T10:23:38.096285 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-03-30T10:23:38.101218 ipmi ipmi_reset KeyError
2019-03-30T10:23:38.102724 ipmi ipmi_cycle KeyError
2019-03-30T10:23:38.104251 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|acraciun
Relops controller action failed:
2019-04-01T06:42:38.487945 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-04-01T06:45:52.735915 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-04-01T06:45:52.742512 ipmi ipmi_reset KeyError
2019-04-01T06:45:52.745399 ipmi ipmi_cycle KeyError
2019-04-01T06:45:52.747553 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|acraciun
Relops controller action failed:
2019-04-01T11:48:47.961330 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-04-01T11:52:02.146015 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-04-01T11:52:02.152772 ipmi ipmi_reset KeyError
2019-04-01T11:52:02.154329 ipmi ipmi_cycle KeyError
2019-04-01T11:52:02.155830 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|zfay
Relops controller action failed:
2019-04-01T21:09:12.993457 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-04-01T21:12:27.216566 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-04-01T21:12:27.219230 ipmi ipmi_reset KeyError
2019-04-01T21:12:27.220521 ipmi ipmi_cycle KeyError
2019-04-01T21:12:27.221825 ipmi ipmi_on KeyError

I cycled the power on this machine and it did not recover.

I powered it off and will leave it off for a few days and then we can check to see if it will come up and work (and to see if it starts failing again after some time after that).

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|acraciun
Relops controller action failed:
2019-04-02T06:47:19.438862 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-04-02T06:50:33.617139 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-04-02T06:50:33.622092 ipmi ipmi_reset KeyError
2019-04-02T06:50:33.624847 ipmi ipmi_cycle KeyError
2019-04-02T06:50:33.626362 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|rmutter
Relops controller action failed:
2019-04-02T19:39:50.807017 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-04-02T19:43:04.980875 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-04-02T19:43:04.986783 ipmi ipmi_reset KeyError
2019-04-02T19:43:04.991188 ipmi ipmi_cycle KeyError
2019-04-02T19:43:04.994407 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|acraciun
Relops controller action failed:
2019-04-03T06:32:42.775564 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-04-03T06:35:56.942516 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-04-03T06:35:56.945259 ipmi ipmi_reset KeyError
2019-04-03T06:35:56.946793 ipmi ipmi_cycle KeyError
2019-04-03T06:35:56.948330 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|acraciun
Relops controller action failed:
2019-04-03T11:52:56.455974 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-04-03T11:56:10.664655 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-04-03T11:56:10.669786 ipmi ipmi_reset KeyError
2019-04-03T11:56:10.673393 ipmi ipmi_cycle KeyError
2019-04-03T11:56:10.675989 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|apop
Relops controller action failed:
2019-04-03T14:12:15.309856 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-04-03T14:15:29.507992 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-04-03T14:15:29.511206 ipmi ipmi_reset KeyError
2019-04-03T14:15:29.512798 ipmi ipmi_cycle KeyError
2019-04-03T14:15:29.514448 ipmi ipmi_on KeyError

Reboot t-yosemite-r7-300.test.releng.mdc1.mozilla.com. 10.49.56.241
Requested by mozilla-auth0/ad|Mozilla-LDAP|dhouse
Relops controller action failed:
2019-04-03T16:05:56.725442 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-04-03T16:09:10.899325 snmp_reboot pdu1.rit44.ops.releng.mdc1.mozilla.com bc15 --delay 60 Exception
2019-04-03T16:09:10.901451 ipmi ipmi_reset KeyError
2019-04-03T16:09:10.902812 ipmi ipmi_cycle KeyError
2019-04-03T16:09:10.904163 ipmi ipmi_on KeyError

powering off for a few days

Status: NEW → RESOLVED
Closed: 7 years ago
Flags: needinfo?(dhouse)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.