Closed Bug 456553 Opened 13 years ago Closed 13 years ago

network issues at the colo

Categories

(mozilla.org Graveyard :: Server Operations, task)

task
Not set
blocker

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: nthomas, Unassigned)

Details

[21:56]	<nagios>	
        bm-xserve16.build is DOWN: CRITICAL - Host Unreachable (10.2.71.113)
[21:57]	qm-moz2mini01 is DOWN: CRITICAL - Plugin timed out after 10 seconds
[21:58]	bm-xserve01.build is DOWN: CRITICAL - Host Unreachable (10.2.71.38)
[21:59]	bm-xserve02.build is DOWN: CRITICAL - Host Unreachable (10.2.71.36)
	bm-xserve17.build is DOWN: CRITICAL - Host Unreachable (10.2.71.114)

(times are GMT+12). I've verified that ssh hangs when trying to open a connection to each host. Of these 5, qm-moz2mini01 is the only mac unit test box for Fx3.1, bm-xserve17 is one of three build slaves for Fx3.1 - both we'd like back asap to not block developers, the others are less critical (but 16 blocks release automation development for Fx3.1b1).

Ause is also having trouble connecting to cb-xserve03.mozilla.com.

Dead network/power gear at the colo ?
We're getting other machines fail now 
[22:25]	<nagios>	
    tbnewref-win32-tbox.build is DOWN: CRITICAL - Host Unreachable (10.2.71.101)
    fx-win32-tbox.build is DOWN: CRITICAL - Host Unreachable (10.2.71.49)
with are both VM's. Plus I'm having trouble reaching various buildbot masters (so there are Talos boxes losing connection and so on).
Severity: critical → blocker
OS: Mac OS X → All
Hardware: Macintosh → All
Summary: 6 macs went AWOL → network issues at the colo
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.