Last Comment Bug 473112 - Build machines have lost network connectivity
: Build machines have lost network connectivity
Product: Graveyard
Classification: Graveyard
Component: Server Operations (show other bugs)
: other
: All All
: -- blocker (vote)
: ---
Assigned To: matthew zeier [:mrz]
: matthew zeier [:mrz]
Depends on:
Blocks: 473126
  Show dependency treegraph
Reported: 2009-01-11 21:50 PST by Nick Thomas [:nthomas]
Modified: 2015-03-12 08:17 PDT (History)
4 users (show)
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Description Nick Thomas [:nthomas] 2009-01-11 21:50:50 PST
Approximately 70 reports from nagios of the form
 [30] is DOWN: CRITICAL - Host Unreachable (

Linux & Windows VMs, xserves, talos slaves, and buildbot masters.
Comment 1 matthew zeier [:mrz] 2009-01-11 22:00:26 PST
Related to switch upgrade which has failed, possibly due to a faulty memory card.  I'm not sure why ESX is failing to connect to storage (spanning-tree has already failed over but the service console is unresponsive).
Comment 2 matthew zeier [:mrz] 2009-01-11 22:03:45 PST
netapp-d is single homed or for some reason hasn't failed over.  I was on the wrong track about ESX - it's fine but any VM on netapp-d is likely offline.
Comment 3 matthew zeier [:mrz] 2009-01-11 22:08:27 PST
See bug 473113.
Comment 4 matthew zeier [:mrz] 2009-01-11 22:43:42 PST
I rebooted fx-win32-tbox.  Can't seem to find anyone from build online for guidance on what's up or down.
Comment 5 Nick Thomas [:nthomas] 2009-01-11 23:09:26 PST
I'll be around in 30 minutes or so to bring machines back up.
Comment 6 Nick Thomas [:nthomas] 2009-01-12 06:08:31 PST
IT fixed this up.
Comment 7 Nick Thomas [:nthomas] 2009-01-12 06:09:50 PST
... and we're tracking bringing the machines back up in bug 473126.

Note You need to log in before you can comment on or make changes to this bug.