1377050 - admin2.private.pao1.mozilla.com (10.34.75.11) is DOWN :PING CRITICAL - Packet loss = 100%

Status: RESOLVED FIXED

(Infrastructure & Operations :: MOC: Problems, task)

Description

[Ryan C [:ryanc] (UTC-4)]

Attachment: Screen Shot 2017-06-28 at 9.56.58 PM.png

The following host went down,

21:22:03 <@nagios-scl3> Thu 04:22:03 UTC [5922] [systems] admin2.private.pao1.mozilla.com (10.34.75.11) is DOWN :PING CRITICAL - Packet loss = 100%

Tried restarting the system with no luck, looks like an HP host,

[[e-mail]1 ~]$ fping admin2.private.pao1.mozilla.com
ICMP Host Unreachable from 10.34.75.5 for ICMP Echo sent to admin2.private.pao1.mozilla.com (10.34.75.11)
ICMP Host Unreachable from 10.34.75.5 for ICMP Echo sent to admin2.private.pao1.mozilla.com (10.34.75.11)
ICMP Host Unreachable from 10.34.75.5 for ICMP Echo sent to admin2.private.pao1.mozilla.com (10.34.75.11)
admin2.private.pao1.mozilla.com is unreachable
[[e-mail]1 ~]$ fping 10.34.75.51
10.34.75.51 is alive
[[e-mail]1 ~]$ ipmitool -H 10.34.75.51 -U ADMIN chassis power reset
Password:
Authentication type NONE not supported
Error: Unable to establish LAN session


Tried to console the host and reset it with no luck, getting the following error when trying to view the console.

Comment 1

[Ryan C [:ryanc] (UTC-4)]

Trying via ilo cli with no luck,


[[e-mail]1 ~]$ ssh mozillaadmin@10.34.75.51
The authenticity of host '10.34.75.51 (10.34.75.51)' can't be established.
RSA key fingerprint is 39:1f:3a:98:13:0d:af:5b:d3:e2:54:e4:db:fc:b3:91.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.34.75.51' (RSA) to the list of known hosts.
mozillaadmin@10.34.75.51's password:
User:mozillaadmin logged-in to ILOMX262100F9.private.mtv2.mozilla.com(10.34.75.51)

iLO Standard 2.50 at  Sep 23 2016
Server Name:
Server Power: On

</>hpiLO-> power reset

status=0
status_tag=COMMAND COMPLETED
Thu Jun 29 05:02:38 2017



Server resetting .......



</>hpiLO-> textcons

status=2
status_tag=COMMAND PROCESSING FAILED
error_tag=COMMAND ERROR-UNSPECIFIED
Thu Jun 29 05:03:02 2017

iLO Advanced License required for this functionality.


</>hpiLO-> vsp

Virtual Serial Port Active: COM1

Starting virtual serial port.
Press 'ESC (' to return to the CLI Session.

Comment 2

[Ryan C [:ryanc] (UTC-4)]

This probably needs to be consoled in person.

Comment 3

[Van Le [:van]]

back online.


[[e-mail]1 ~]$ fping admin2.private.pao1.mozilla.com
admin2.private.pao1.mozilla.com is alive
[[e-mail]1 ~]$ ssh admin2.private.pao1.mozilla.com
The authenticity of host 'admin2.private.pao1.mozilla.com (10.34.75.11)' can't be established.
RSA key fingerprint is 74:3b:40:ad:01:cb:2f:dd:f1:2b:8a:eb:38:03:3f:31.
Are you sure you want to continue connecting (yes/no)?

Comment 4

[Ryan C [:ryanc] (UTC-4)]

Curious what was done to resolve this - Thanks

Comment 5

[Van Le [:van]]

i power cycled the server from iLo and watched it boot up. didn't see any issues or errors during boot up.