mw32-ix-slave* slaves lost connection to master

RESOLVED FIXED

Status

mozilla.org Graveyard
Server Operations
P1
blocker
RESOLVED FIXED
8 years ago
3 years ago

People

(Reporter: Away for a while, Assigned: phong)

Tracking

Details

I see 8 failures, across several mozilla-central and jaegermonkey. The common factor is win32 ix hardware, mw32-ix-slave*.
Assignee: nobody → nrthomas
Priority: -- → P1
Summary: mozilla-central jobs failing with type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'callRemote' → mw32-ix-slave* slaves lost connection to master
3 were at 15:44, 5 more at 14:35; two on pm03, 6 on pm01 - can't find anything wrong with the maters. Seems like this would be a switch or network fault if it affects such a specific group of machines.
From twistd.log on mw32-ix-slave19:

2010/08/31 15:34 -0700 [Broker,client] SlaveBuilder._ackFailed: SlaveBuilder.sendUpdate
[repeats 6 more times]
2010/08/31 15:34 -0700 [Broker,client] lost remote
[repeats many more times]
2010/08/31 15:34 -0700 [Broker,client] lost remote step
2010/08/31 15:34 -0700 [Broker,client] stopCommand: halting current command <buildbot.slave.commands.base.SlaveShellCommand instance at 0x014521C0>

dmoore says the network between Castro and MPT is very busy at the moment, so the latency is causing the slave to drop the connection to the master, or we're dropping connections in some network hardware. --> IT.

I have no problem at all with throttling the xserve archiving that is chewing bandwidth.
Assignee: nrthomas → server-ops
Component: Release Engineering → Server Operations
QA Contact: release → mrz
(Reporter)

Comment 4

8 years ago
(In reply to comment #3)
> I have no problem at all with throttling the xserve archiving that is chewing
> bandwidth.

OK, can you do that?  If yes, then will you reopen the tree please?
Phong, can you restart the rsync with a cap of around 10 Mbps or so?
(Assignee)

Comment 6

8 years ago
We've throttled down my copy.  This should be fixed now.
Assignee: server-ops → phong
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
Thanks Phong. Tree reopened.
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.