Closed Bug 1472842 Opened 7 years ago Closed 7 years ago

t-yosemite-r7-260.test.releng.mdc1.mozilla.com. is unreachable

Categories

(Infrastructure & Operations :: DCOps, task)

Product:
Infrastructure & Operations
Component:
DCOps
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: relops-bug-generator, Assigned: van)

References

Details

(Whiteboard: REQ0235361, REQ0235523 )

Reboot t-yosemite-r7-260.test.releng.mdc1.mozilla.com. 10.49.56.201 Requested by mozilla-auth0/ad|Mozilla-LDAP|riman Relops controller action failed: 2018-07-02T22:35:08.557489 ssh_reboot -l roller -i ssh.key TimeoutExpired 2018-07-02T22:35:08.561584 ipmi ipmi_reset KeyError 2018-07-02T22:35:08.564633 ipmi ipmi_cycle KeyError
power cycling didn't work, opened REQ0235361 with QTS for remote hands.
Assignee: server-ops-dcops → vle
Whiteboard: REQ0235361
Reboot t-yosemite-r7-260.test.releng.mdc1.mozilla.com. 10.49.56.201 Requested by mozilla-auth0/ad|Mozilla-LDAP|zfay Relops controller action failed: 2018-07-03T09:31:46.243677 ssh_reboot -l roller -i ssh.key TimeoutExpired 2018-07-03T09:31:46.247844 ipmi ipmi_reset KeyError 2018-07-03T09:31:46.250721 ipmi ipmi_cycle KeyError
opened REQ0235523 with QTS to reimage mini.
Whiteboard: REQ0235361 → REQ0235361, REQ0235523
The machine seems to be down. I've tried to ping it and it gave me ping timed out.
Reboot t-yosemite-r7-260.test.releng.mdc1.mozilla.com. 10.49.56.201 Requested by mozilla-auth0/ad|Mozilla-LDAP|zfay Relops controller action failed: 2018-07-08T12:45:25.022363 ssh_reboot -l roller -i ssh.key TimeoutExpired 2018-07-08T12:45:25.024566 ipmi ipmi_reset KeyError 2018-07-08T12:45:25.026570 ipmi ipmi_cycle KeyError
qts task completed, host should have reimaged. let me know if issues persist.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
This machine has the same problem again. Unreachable through SSH.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Reboot t-yosemite-r7-260.test.releng.mdc1.mozilla.com. 10.49.56.201 Requested by mozilla-auth0/ad|Mozilla-LDAP|dhouse Relops controller action failed: 2018-07-23T18:36:37.987212 ssh_reboot -l roller -i ssh.key TimeoutExpired 2018-07-23T18:41:47.991061 ipmi ipmi_reset KeyError 2018-07-23T18:41:47.995275 ipmi ipmi_cycle KeyError
No response to ping/ssh and cycling with snmp does not repair it. Also, with snmp I think it is not powering-off (power status stays on when I've asked it to turn off): ``` # snmpget -v 2c -c public pdu1.rit44.ops.releng.mdc1.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.5.2.2.7 iso.3.6.1.4.1.1718.3.2.3.1.5.2.2.7 = INTEGER: 1 # snmpset -v 2c -c secret pdu1.rit41.ops.releng.mdc1.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.11.2.2.7 i 2 iso.3.6.1.4.1.1718.3.2.3.1.11.2.2.7 = INTEGER: 2 # snmpget -v 2c -c public pdu1.rit44.ops.releng.mdc1.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.5.2.2.7 iso.3.6.1.4.1.1718.3.2.3.1.5.2.2.7 = INTEGER: 1 # snmpget -v 2c -c public pdu1.rit44.ops.releng.mdc1.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.5.2.2.7 iso.3.6.1.4.1.1718.3.2.3.1.5.2.2.7 = INTEGER: 1 # snmpset -v 2c -c secret pdu1.rit41.ops.releng.mdc1.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.11.2.2.7 i 3 iso.3.6.1.4.1.1718.3.2.3.1.11.2.2.7 = INTEGER: 3 # snmpget -v 2c -c public pdu1.rit44.ops.releng.mdc1.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.5.2.2.7 iso.3.6.1.4.1.1718.3.2.3.1.5.2.2.7 = INTEGER: 1 # snmpget -v 2c -c public pdu1.rit44.ops.releng.mdc1.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.5.2.2.7 iso.3.6.1.4.1.1718.3.2.3.1.5.2.2.7 = INTEGER: 1 # snmpset -v 2c -c secret pdu1.rit41.ops.releng.mdc1.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.11.2.2.7 i 2 iso.3.6.1.4.1.1718.3.2.3.1.11.2.2.7 = INTEGER: 2 # snmpget -v 2c -c public pdu1.rit44.ops.releng.mdc1.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.5.2.2.7 iso.3.6.1.4.1.1718.3.2.3.1.5.2.2.7 = INTEGER: 1 # snmpget -v 2c -c public pdu1.rit44.ops.releng.mdc1.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.5.2.2.7 iso.3.6.1.4.1.1718.3.2.3.1.5.2.2.7 = INTEGER: 1 # snmpset -v 2c -c secret pdu1.rit41.ops.releng.mdc1.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.11.2.2.7 i 1 iso.3.6.1.4.1.1718.3.2.3.1.11.2.2.7 = INTEGER: 1 # snmpget -v 2c -c public pdu1.rit44.ops.releng.mdc1.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.5.2.2.7 iso.3.6.1.4.1.1718.3.2.3.1.5.2.2.7 = INTEGER: 1 # snmpget -v 2c -c public pdu1.rit44.ops.releng.mdc1.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.5.2.2.7 iso.3.6.1.4.1.1718.3.2.3.1.5.2.2.7 = INTEGER: 1 ```
should be good after reimage. vle@DESKTOP-3HK51T3:~$ fping t-yosemite-r7-260.test.releng.mdc1.mozilla.com t-yosemite-r7-260.test.releng.mdc1.mozilla.com is alive
Status: REOPENED → RESOLVED
Closed: 7 years ago → 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.