Closed Bug 1478529 Opened 7 years ago Closed 7 years ago

t-yosemite-r7-334.test.releng.mdc1.mozilla.com. is unreachable

Categories

(Infrastructure & Operations :: DCOps, task)

Product:
Infrastructure & Operations
Component:
DCOps
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: relops-bug-generator, Assigned: van)

References

Details

Reboot t-yosemite-r7-334.test.releng.mdc1.mozilla.com. 10.49.56.118 Requested by mozilla-auth0/ad|Mozilla-LDAP|zfay Relops controller action failed: 2018-07-26T00:35:31.063232 ssh_reboot -l roller -i ssh.key TimeoutExpired 2018-07-26T00:40:41.060967 ipmi ipmi_reset KeyError 2018-07-26T00:40:41.062922 ipmi ipmi_cycle KeyError
Tried to ssh into the machine and it just hangs. Pinged it and got this: zfay@zsolt:~$ ping t-yosemite-r7-334.test.releng.mdc1.mozilla.com PING t-yosemite-r7-334.test.releng.mdc1.mozilla.com (10.49.56.118) 56(84) bytes of data. --- t-yosemite-r7-334.test.releng.mdc1.mozilla.com ping statistics --- 15 packets transmitted, 0 received, 100% packet loss, time 14330ms
Reboot t-yosemite-r7-334.test.releng.mdc1.mozilla.com. 10.49.56.118 Requested by mozilla-auth0/ad|Mozilla-LDAP|dhouse Relops controller action failed: 2018-07-27T20:26:16.658373 ssh_reboot -l roller -i ssh.key TimeoutExpired 2018-07-27T20:31:11.632999 snmp_reboot pdu1.rit40.ops.releng.mdc1.mozilla.com ba4 SoftTimeLimitExceeded 2018-07-27T20:31:11.636467 ipmi ipmi_reset KeyError 2018-07-27T20:31:11.638581 ipmi ipmi_cycle KeyError
I tried manually with snmp powering this machine off and then back on, and still no response to ping (and no ssh). Please physically reboot and check that this machine is booting up. ``` # snmpget -v 2c -c public pdu1.rit40.ops.releng.mdc1.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.5.2.1.4 iso.3.6.1.4.1.1718.3.2.3.1.5.2.1.4 = INTEGER: 1 # snmpset -v 2c -c secret pdu1.rit40.ops.releng.mdc1.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.11.2.1.4 i 2 iso.3.6.1.4.1.1718.3.2.3.1.11.2.1.4 = INTEGER: 2 # snmpget -v 2c -c public pdu1.rit40.ops.releng.mdc1.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.5.2.1.4 iso.3.6.1.4.1.1718.3.2.3.1.5.2.1.4 = INTEGER: 0 # snmpset -v 2c -c secret pdu1.rit40.ops.releng.mdc1.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.11.2.1.4 i 1 iso.3.6.1.4.1.1718.3.2.3.1.11.2.1.4 = INTEGER: 1 # snmpget -v 2c -c public pdu1.rit40.ops.releng.mdc1.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.5.2.1.4 iso.3.6.1.4.1.1718.3.2.3.1.5.2.1.4 = INTEGER: 1 [dhouse@rejh2.srv.releng.mdc1.mozilla.com ~]$ ping t-yosemite-r7-334.test.releng.mdc1.mozilla.com PING t-yosemite-r7-334.test.releng.mdc1.mozilla.com (10.49.56.118) 56(84) bytes of data. ^C --- t-yosemite-r7-334.test.releng.mdc1.mozilla.com ping statistics --- 112 packets transmitted, 0 received, 100% packet loss, time 111345ms ```
back online after reimage. [vle@admin2a.private.mdc1 ~]$ fping t-yosemite-r7-334.test.releng.mdc1.mozilla.com t-yosemite-r7-334.test.releng.mdc1.mozilla.com is alive
Assignee: server-ops-dcops → vle
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.