658263 - w32: runslave.py gets SIGBREAK on startup

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

From w32-ix-slave01:

"Mozilla tools directory: D:\mozilla-build\"
Windows SDK directory: D:\sdks\v7.0\
Windows SDK version: 7.0
Setting environment for using Microsoft Visual Studio 2005 x86 tools.
"Wed 05/18/2011 14:52:57.29 - About to run runslave.py"
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:201: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:202: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:202: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:263: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:266: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:270: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:275: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:280: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:297: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:299: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:300: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:303: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:365: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:387: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:458: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:467: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:539: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:543: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:544: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:544: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:547: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:554: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:587: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:605: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:625: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:628: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:633: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:635: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:636: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:637: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:660: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:662: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:666: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:667: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:671: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:672: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:702: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:705: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:715: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:716: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:724: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:727: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:737: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:738: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:743: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:743: Warni
ng: 'with' will become a reserved keyword in Python 2.6
D:\mozilla-build\python25\lib\site-packages\zope\interface\adapter.py:746: Warni
ng: 'with' will become a reserved keyword in Python 2.6
2011/05/18 14:53 -0700 [-] Log opened.
2011/05/18 14:53 -0700 [-] twistd 2.5.0 (D:\mozilla-build\python25\python.exe 2.
5.0) starting up
2011/05/18 14:53 -0700 [-] reactor class: <class 'twisted.internet.selectreactor
.SelectReactor'>
2011/05/18 14:53 -0700 [-] Loading e:\builds\moz2_slave\buildbot.tac...
2011/05/18 14:53 -0700 [-] Creating BuildSlave -- buildbot.version: 0.7.10p1
2011/05/18 14:53 -0700 [-] Loaded.
2011/05/18 14:53 -0700 [-] Starting factory <buildbot.slave.bot.BotFactory insta
nce at 0x016BE508>
2011/05/18 14:53 -0700 [Broker,client] message from master: attached
2011/05/18 14:53 -0700 [Broker,client] Peer will receive following PB traceback:

2011/05/18 14:53 -0700 [Broker,client] Unhandled Error
        Traceback (most recent call last):
          File "D:\mozilla-build\python25\lib\site-packages\twisted\spread\banan
a.py", line 148, in gotItem
            self.callExpressionReceived(item)
          File "D:\mozilla-build\python25\lib\site-packages\twisted\spread\banan
a.py", line 113, in callExpressionReceived
            self.expressionReceived(obj)
          File "D:\mozilla-build\python25\lib\site-packages\twisted\spread\pb.py
", line 522, in expressionReceived
            method(*sexp[1:])
          File "D:\mozilla-build\python25\lib\site-packages\twisted\spread\pb.py
", line 833, in proto_message
            self._recvMessage(self.localObjectForID, requestID, objectID, messag
e, answerRequired, netArgs, netKw)
        --- <exception caught here> ---
          File "D:\mozilla-build\python25\lib\site-packages\twisted\spread\pb.py
", line 847, in _recvMessage
            netResult = object.remoteMessageReceived(self, message, netArgs, net
Kw)
          File "d:\mozilla-build\python25\Lib\site-packages\twisted\spread\flavo
rs.py", line 117, in remoteMessageReceived
            raise NoSuchMethod("No such method: remote_%s" % (message,))
        twisted.spread.flavors.NoSuchMethod: No such method: remote_getVersion

2011/05/18 14:53 -0700 [Broker,client] Received SIGBREAK, shutting down.
2011/05/18 14:53 -0700 [Broker,client] I have a leftover directory 'bir-w32-dbg'
 that is not being used by the buildmaster: you can delete it now
2011/05/18 14:53 -0700 [Broker,client] ReconnectingPBClientFactory.failedToGetPe
rspective
2011/05/18 14:53 -0700 [Broker,client] we lost the brand-new connection
2011/05/18 14:53 -0700 [Broker,client] Stopping factory <buildbot.slave.bot.BotF
actory instance at 0x016BE508>
2011/05/18 14:53 -0700 [-] Main loop terminated.
2011/05/18 14:53 -0700 [-] Server Shut Down.
"Wed 05/18/2011 14:53:02.67 - runslave.py finished"

D:\mozilla-build>

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

This only failed on one of the three {m,}w32-ix-slaveNN machines in staging, and none of the three win32-slaveNN's

Here's a relevant thread:

http://twistedmatrix.com/pipermail/twisted-python/2006-September/014111.html

All of these slaves have Twisted-2.5.0 on them, which is from around 2006, so that makes sense.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

It doesn't look like this has been changed in more recent Twisted versions.  A few guesses:

buildbot.bat's |START /MIN start-buildbot.bat| starts the script minimized - perhaps this triggers a signal?  Too bad, because tests assume an empty desktop.

buildbot.bat exits right after invoking START, so perhaps the SIGBREAK for buildbot.bat is affecting start-buildbot.bat as well, in a race-conditiony way (this would explain why the failure happens more often on iX machines).

at any rate, a few fixes:

1. sleep for 30s after the START

2. try to make the Twisted reactor immune to SIGBREAK by monkeypatching reactor.sigBreak to ignore the signal.

I doubt the latter would work well, since if SIGBREAK is received before the signal handlers are installed, the default action is to exit.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: m658263-opsi-package-sources-r1.patch

Option (1) seems to have done the trick, over a number of reboots on the three staging {m,}w32-ix-slaveNN machines.