Approximately 70 reports from nagios of the form
 moz2-linux-slave04.build is DOWN: CRITICAL - Host Unreachable (10.2.71.18)
Linux & Windows VMs, xserves, talos slaves, and buildbot masters.
Related to switch upgrade which has failed, possibly due to a faulty memory card. I'm not sure why ESX is failing to connect to storage (spanning-tree has already failed over but the service console is unresponsive).
netapp-d is single homed or for some reason hasn't failed over. I was on the wrong track about ESX - it's fine but any VM on netapp-d is likely offline.
See bug 473113.
I rebooted fx-win32-tbox. Can't seem to find anyone from build online for guidance on what's up or down.
I'll be around in 30 minutes or so to bring machines back up.
IT fixed this up.
... and we're tracking bringing the machines back up in bug 473126.