456553 - network issues at the colo

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

[21:56]	<nagios>	
        bm-xserve16.build is DOWN: CRITICAL - Host Unreachable (10.2.71.113)
[21:57]	qm-moz2mini01 is DOWN: CRITICAL - Plugin timed out after 10 seconds
[21:58]	bm-xserve01.build is DOWN: CRITICAL - Host Unreachable (10.2.71.38)
[21:59]	bm-xserve02.build is DOWN: CRITICAL - Host Unreachable (10.2.71.36)
	bm-xserve17.build is DOWN: CRITICAL - Host Unreachable (10.2.71.114)

(times are GMT+12). I've verified that ssh hangs when trying to open a connection to each host. Of these 5, qm-moz2mini01 is the only mac unit test box for Fx3.1, bm-xserve17 is one of three build slaves for Fx3.1 - both we'd like back asap to not block developers, the others are less critical (but 16 blocks release automation development for Fx3.1b1).

Ause is also having trouble connecting to cb-xserve03.mozilla.com.

Dead network/power gear at the colo ?

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

We're getting other machines fail now 
[22:25]	<nagios>	
    tbnewref-win32-tbox.build is DOWN: CRITICAL - Host Unreachable (10.2.71.101)
    fx-win32-tbox.build is DOWN: CRITICAL - Host Unreachable (10.2.71.49)
with are both VM's. Plus I'm having trouble reaching various buildbot masters (so there are Talos boxes losing connection and so on).