Closed Bug 515348 Opened 16 years ago Closed 16 years ago

slaves in the new vlan sometimes lose their remote

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Platform:
x86
All
Type:
task
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: bhearsum, Assigned: dmoore)

References

Details

I was having a look at the balance of the slave pool and noticed that many (2 linux, 7 win32) slaves that are in the new vlan lost their remote, indicated by this in the Buildbot slave log (also by their absence on the master): 2009/09/09 04:31 -0700 [Broker,client] lost remote 2009/09/09 04:31 -0700 [Broker,client] lost remote 2009/09/09 04:31 -0700 [Broker,client] lost remote 2009/09/09 04:31 -0700 [Broker,client] lost remote 2009/09/09 04:31 -0700 [Broker,client] lost remote 2009/09/09 04:31 -0700 [Broker,client] lost remote 2009/09/09 04:31 -0700 [Broker,client] lost remote 2009/09/09 04:31 -0700 [Broker,client] lost remote 2009/09/09 04:31 -0700 [Broker,client] lost remote 2009/09/09 04:31 -0700 [Broker,client] lost remote A few theories off the top of my head: * too many slaves on one master is making Buildbot lose some * the new vlan is configured in such a way that it's causing this * the new slaves somehow have a different version of buildbot that's busted in some way Given that this problem hasn't shown up until the new vlan existed, and has only happened on the new slaves it seems likely that it's option #2 or #3. We should do a buildbot version comparison on a few new slaves vs old to look at that possibility, and rope in IT to look at the vlan config.
happened to linux-slave33 today
linux-slave32 and linux-slave34 today
win32-slave43 and 45, too.
moz2-linux-slave27 (vlan 90) http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1252961316.1252965708.702.gz IT, please investigate what could be causing network interruptions between our buildbot masters in vlan71 and newer slaves in vlan90. Are any ESX hosts in vlan90 ? Are those hosts correctly configured for both vlans ?
Assignee: nobody → server-ops
Severity: normal → major
Component: Release Engineering → Server Operations
OS: Mac OS X → All
QA Contact: release → mrz
Assignee: server-ops → dmoore
are both DHCP servers (and the hosting ESX's switch ports in the case of nm-dhcp01) correctly configured for both vlans?
Ben/Nick, Our internal firewall will terminate TCP connections which have been idle for longer than 60 minutes. I suspect this is the issue, and I have removed this timeout for build<->build traffic. Please let me know if you encounter another interruption and we'll investigate further.
Per Ben, no broken connections since the timeout was lifted. Marking as resolved.
Status: NEW → RESOLVED
Closed: 16 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.