580170 - notorious aka lists.mozilla.org is down

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Shyam Mani [:fox2mike]]

We seem to have lost the machine and it isn't reachable on iLO either.

21:25:04 <@nagios> [20] notorious:SMTP is CRITICAL: CRITICAL - Socket timeout after 10 seconds
21:25:17 <@nagios> [21] notorious:RAID is CRITICAL: (Service Check Timed Out)
21:25:35 <@nagios> [22] notorious:https - lists.mozilla.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds
21:25:54 <@nagios> [23] notorious:http - lists.mozilla.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds
21:26:08 <@nagios> [24] notorious:SMTP cert - port 25 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
21:31:12 <@nagios> [26] notorious:root partition is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.
21:32:57 <@nagios> [27] notorious:avg load is CRITICAL: CHECK_NRPE: Socket timeout after 15 seconds.

Comment 1

[Shyam Mani [:fox2mike]]

Since iLO is down, we can't do much remotely. Might need someone to go to the datacenter and have a look :|

Comment 2

[Phong Tran [:phong]]

I will go first thing in the morning unless this needs attention right away.

Comment 3

[Phong Tran [:phong]]

changed iLO to static.  The servers is up and running again.