admin2.private.pao1.mozilla.com (10.34.75.11) is DOWN :PING CRITICAL - Packet loss = 100%

RESOLVED FIXED

Status

Infrastructure & Operations
MOC: Problems
RESOLVED FIXED
7 months ago
7 months ago

People

(Reporter: ryanc, Assigned: van)

Tracking

Details

Attachments

(1 attachment)

(Reporter)

Description

7 months ago
Created attachment 8882083 [details]
Screen Shot 2017-06-28 at 9.56.58 PM.png

The following host went down,

21:22:03 <@nagios-scl3> Thu 04:22:03 UTC [5922] [systems] admin2.private.pao1.mozilla.com (10.34.75.11) is DOWN :PING CRITICAL - Packet loss = 100%

Tried restarting the system with no luck, looks like an HP host,

[rchilds@admin1.private.pao1 ~]$ fping admin2.private.pao1.mozilla.com
ICMP Host Unreachable from 10.34.75.5 for ICMP Echo sent to admin2.private.pao1.mozilla.com (10.34.75.11)
ICMP Host Unreachable from 10.34.75.5 for ICMP Echo sent to admin2.private.pao1.mozilla.com (10.34.75.11)
ICMP Host Unreachable from 10.34.75.5 for ICMP Echo sent to admin2.private.pao1.mozilla.com (10.34.75.11)
admin2.private.pao1.mozilla.com is unreachable
[rchilds@admin1.private.pao1 ~]$ fping 10.34.75.51
10.34.75.51 is alive
[rchilds@admin1.private.pao1 ~]$ ipmitool -H 10.34.75.51 -U ADMIN chassis power reset
Password:
Authentication type NONE not supported
Error: Unable to establish LAN session


Tried to console the host and reset it with no luck, getting the following error when trying to view the console.
(Reporter)

Comment 1

7 months ago
Trying via ilo cli with no luck,


[rchilds@admin1.private.pao1 ~]$ ssh mozillaadmin@10.34.75.51
The authenticity of host '10.34.75.51 (10.34.75.51)' can't be established.
RSA key fingerprint is 39:1f:3a:98:13:0d:af:5b:d3:e2:54:e4:db:fc:b3:91.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.34.75.51' (RSA) to the list of known hosts.
mozillaadmin@10.34.75.51's password:
User:mozillaadmin logged-in to ILOMX262100F9.private.mtv2.mozilla.com(10.34.75.51)

iLO Standard 2.50 at  Sep 23 2016
Server Name:
Server Power: On

</>hpiLO-> power reset

status=0
status_tag=COMMAND COMPLETED
Thu Jun 29 05:02:38 2017



Server resetting .......



</>hpiLO-> textcons

status=2
status_tag=COMMAND PROCESSING FAILED
error_tag=COMMAND ERROR-UNSPECIFIED
Thu Jun 29 05:03:02 2017

iLO Advanced License required for this functionality.


</>hpiLO-> vsp

Virtual Serial Port Active: COM1

Starting virtual serial port.
Press 'ESC (' to return to the CLI Session.
(Reporter)

Comment 2

7 months ago
This probably needs to be consoled in person.
Flags: needinfo?(vle)
(Reporter)

Updated

7 months ago
Summary: 21:22:03 <@nagios-scl3> Thu 04:22:03 UTC [5922] [systems] admin2.private.pao1.mozilla.com (10.34.75.11) is DOWN :PING CRITICAL - Packet loss = 100% → admin2.private.pao1.mozilla.com (10.34.75.11) is DOWN :PING CRITICAL - Packet loss = 100%
(Assignee)

Comment 3

7 months ago
back online.


[vle@admin1.private.pao1 ~]$ fping admin2.private.pao1.mozilla.com
admin2.private.pao1.mozilla.com is alive
[vle@admin1.private.pao1 ~]$ ssh admin2.private.pao1.mozilla.com
The authenticity of host 'admin2.private.pao1.mozilla.com (10.34.75.11)' can't be established.
RSA key fingerprint is 74:3b:40:ad:01:cb:2f:dd:f1:2b:8a:eb:38:03:3f:31.
Are you sure you want to continue connecting (yes/no)?
Assignee: nobody → vle
Status: NEW → RESOLVED
Last Resolved: 7 months ago
Flags: needinfo?(vle)
Resolution: --- → FIXED
(Reporter)

Comment 4

7 months ago
Curious what was done to resolve this - Thanks
(Assignee)

Comment 5

7 months ago
i power cycled the server from iLo and watched it boot up. didn't see any issues or errors during boot up.
You need to log in before you can comment on or make changes to this bug.