723299 - hg outage - Feb 1 2012

Status: RESOLVED INCOMPLETE

(mozilla.org Graveyard :: Server Operations, task)

Description

[John Ford [:jhford] CET/CEST Berlin Time]

[46] dm-hg02:http - hg.mozilla.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds
nagios-sjc1
[48] dm-hg02:https_cert - hg.mozilla.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds
nagios-sjc1
[50] dm-svn02:health is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.
nagios-sjc1
[52] dm-hg02:https - hg.mozilla.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds
nagios-sjc1
[54] dm-svn02:root partition is CRITICAL: CHECK_NRPE: Socket timeout after 60 seconds.

Looks like HG is down.

Comment 1

[Ben Kero [:bkero]]

This was a general host failure of dm-(svn|hg)02.  The machine has been rebooted and services are coming back online.

Comment 2

[Ben Kero [:bkero]]

I've checked the log files, and cannot find anything indicating a failure, or even a graceful shutdown.  This along with the following description lead me to believe it was disk read failure.

14:51 <@justdave> I was using the java console
14:51 <@justdave> there was a banner and a login prompt, and it echoed back when I typed in "root<enter>"
14:51 <@justdave> and then never gave me a password prompt, and never re-issued the login  prompt

Comment 3

[Ben Kero [:bkero]]

I'm afraid we weren't able to receive any information as to why this happened.  All ILO and system logs told us no clues about why this happened.  If this problem occurs again we can investigate replacing the hardware.