Closed Bug 505153 Opened 15 years ago Closed 15 years ago

non-build-vlan VMs on bm-vmware08 down (eg talos-master & sm-try-master)

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Platform:
x86
Linux
Type:
task
Priority:
Not set
Severity:
blocker

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: mozilla, Assigned: phong)

Details

VI reports that sm-try-master was migrated to bm-vmware08 at ~4:27am.
Nagios reports that sm-try-master became unavailable at ~4:30am.

The MAC address appears to have changed at some point. Either way, I'm unable to either get a DHCP lease or set a static IP via the VI console.  I'm guessing it's either related to the MAC address change or the migration to bm-vmware08 or both.

I've left the static IP settings in /etc/sysconfig/network-scripts/ifcfg-eth0 but left backups in borked-ifcfg-eth0 (and there's an original-ifcfg-eth0 dated jul6)
I've taken this opportunity to update VMWare Tools (to avoid a downtime later).
I will take a look when I get home.
Assignee: server-ops → phong
That doesn't seem to have helped any. On trying to restart the network you get

Bringing up interface eth0:
Determining IP information for eth0...PING 10.2.76.1 (10.2.76.1) from 10.2.76.34 eth0: 56(84) bytes of data.

--- 10.2.76.1 ping statistics ---
3 packes transmitted, 0 received, +3 errors, 100% packet loss, time 1999ms, pipe 3
 failed.


Perhaps a network config problem for the newly setup bm-vmware08 ?
It's a bigger problem than just sm-try-master, seems to be any VM on bm-vmware08 that's not on the build VLAN. Which is  (using VM names from VI)
 talos-master
 try-linux-slave16
 try-master
 try-win32-slave16

So that's two important buildbot masters that are down (no try server or talos pool coverage) --> BLOCKER.
Severity: critical → blocker
Summary: sm-try-master down → non-build-vlan VMs on bm-vmware08 down (eg talos-master & sm-try-master)
try-master and talos-master have been migrated to bm-vmware05 and 04 respectively (but DRS might bring them back again).
Phong - switch config was missing Vlan73 and 76.  Added.
I can reach try-linux-slave16 & try-win32-slave16 now (which are still on bm-vmware08), so this RESOLVED FIXED ?
left it for phong to double check before closing.
everything looks good.
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.