Build machines have lost network connectivity

RESOLVED FIXED

Status

mozilla.org Graveyard
Server Operations
--
blocker
RESOLVED FIXED
9 years ago
2 years ago

People

(Reporter: nthomas, Assigned: mrz)

Tracking

Details

(Reporter)

Description

9 years ago
Approximately 70 reports from nagios of the form
 [30] moz2-linux-slave04.build is DOWN: CRITICAL - Host Unreachable (10.2.71.18)

Linux & Windows VMs, xserves, talos slaves, and buildbot masters.
(Assignee)

Comment 1

9 years ago
Related to switch upgrade which has failed, possibly due to a faulty memory card.  I'm not sure why ESX is failing to connect to storage (spanning-tree has already failed over but the service console is unresponsive).
Assignee: mrz → dmoore
(Assignee)

Comment 2

9 years ago
netapp-d is single homed or for some reason hasn't failed over.  I was on the wrong track about ESX - it's fine but any VM on netapp-d is likely offline.
(Assignee)

Comment 3

9 years ago
See bug 473113.
(Assignee)

Comment 4

9 years ago
I rebooted fx-win32-tbox.  Can't seem to find anyone from build online for guidance on what's up or down.
(Reporter)

Comment 5

9 years ago
I'll be around in 30 minutes or so to bring machines back up.
(Reporter)

Comment 6

9 years ago
IT fixed this up.
Assignee: dmoore → mrz
Status: ASSIGNED → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → FIXED
(Reporter)

Comment 7

9 years ago
... and we're tracking bringing the machines back up in bug 473126.
Blocks: 473126
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.