563931 - mw32-ix-slave17 got in a weird state when running count_and_reboot.py

Status: RESOLVED WORKSFORME

(Release Engineering :: General, defect, P3)

Description

[[e-mail] (:bhearsum)]

mw32-ix-slave17 got a very weird exception in its twistd.log.

2010/04/21 20:32 -0700 [-] trying process.signalProcess('KILL')
2010/04/21 20:32 -0700 [-]  signal KILL sent successfully
2010/04/21 20:32 -0700 [-] command finished with signal None, exit code 1, elapsedTime: 2400.516000
2010/04/21 20:32 -0700 [-] SlaveBuilder.commandComplete <buildbot.slave.commands.SlaveShellCommand instance at 0x019B07B0>
2010/04/21 20:32 -0700 [Broker,client]  startCommand:shell [id 1090568]
2010/04/21 20:32 -0700 [Broker,client] ShellCommand._startCommand
2010/04/21 20:32 -0700 [Broker,client]  python tools/buildfarm/maintenance/count_and_reboot.py -f ../reboot_count.txt -n 1 -z
2010/04/21 20:32 -0700 [Broker,client]   in dir e:\builds\moz2_slave\addonsmgr-win32-debug\. (timeout 1200 secs)
2010/04/21 20:32 -0700 [Broker,client]   watching logfiles {}
2010/04/21 20:32 -0700 [Broker,client]   argv: ['python', 'tools/buildfarm/maintenance/count_and_reboot.py', '-f', '../reboot_count.txt', '-n', '1', '-z']
2010/04/21 20:32 -0700 [Broker,client]  environment: {'LIB': 'D:\\sdks\\v7.0\\\\lib;D:\\msvs8\\VC\\ATLMFC\\LIB;D:\\msvs8\\VC\\LIB;D:\\msvs8\\VC\\PlatformSDK\\lib;D:\\msvs8\\SDK\\v2.0\\lib;;D:\\mozilla-build\\\\atlthunk_compat', 'SYSTEMROOT': 'C:\\WINDOWS', 'APR_ICONV_PATH': 'd:/mozilla-build/svn-win32-1.6.3/iconv', 'SHELL': 'D:/mozilla-build/msys/bin/sh', 'MOZILLABUILDDRIVE': 'D:', 'SDK6KEY': 'HKLM\\SOFTWARE\\Microsoft\\Microsoft SDKs\\Windows\\v6.0', 'JAVA_HOME': 'd:/jdk-1.5.0_10', 'MSVCROOTKEY': 'HKLM\\SOFTWARE\\Microsoft\\VisualStudio', 'COMSPEC': 'C:\\WINDOWS\\system32\\cmd.exe', 'BUILDSLAVE_PASSWORD': '"secret"', 'MSVC71KEY': 'HKLM\\SOFTWARE\\Microsoft\\VisualStudio\\7.1\\Setup\\VC', 'WINCURVERKEY': 'HKLM\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion', 'HOMEDRIVE': 'C:', 'SYSTEMDRIVE': 'C:', 'HOSTNAME': 'mw32-ix-slave17', 'PROCESSOR_LEVEL': '6', 'NVAPSDK': 'D:\\sdks\\tegra042\\', 'MACHTYPE': 'i686-pc-msys', 'OS': 'Windows_NT', 'SDK2003SP1KEY': 'HKLM\\SOFTWARE\\Microsoft\\MicrosoftSDK\\InstalledSDKs\\8F9E5EF3-A9A5-491B-A889-C58EFFECE8B3', 'MSVC9EXPRESSKEY': 'HKLM\\SOFTWARE\\Microsoft\\VCExpress\\9.0\\Setup\\VC', 'USESDK': '1', 'CVS_RSH': 'ssh', 'INPUTRC': 'D:/mozilla-build/msys/etc/inputrc', 'SDK7KEY': 'HKLM\\SOFTWARE\\Microsoft\\Microsoft SDKs\\Windows\\v7.0', 'MSYSTEM': 'MINGW32', 'TEMP': 'C:/DOCUME~1/cltbld/LOCALS~1/Temp', '!EXITCODE': '00000001', 'SHLVL': '2', 'TBOX_CLIENT_CVS_DIR': 'e:/builds/tinderbox/mozilla/tools/tinderbox', 'EDITOR': 'emacs.exe --no-window-system', 'HOMEPATH': '\\', 'VS80COMNTOOLS': 'D:\\msvs8\\Common7\\Tools\\', 'MOZILLABUILD': 'D:\\mozilla-build\\', 'CLUSTERLOG': 'C:\\WINDOWS\\Cluster\\cluster.log', 'LOGONSERVER': '\\\\MW32-IX-SLAVE17', 'INCLUDE': 'D:\\sdks\\v7.0\\\\include;D:\\sdks\\v7.0\\\\include\\atl;D:\\msvs8\\VC\\ATLMFC\\INCLUDE;D:\\msvs8\\VC\\INCLUDE;D:\\msvs8\\VC\\PlatformSDK\\include;', 'CLIENTNAME': 'Console', 'APPDATA': 'C:\\Documents and Settings\\cltbld\\Application Data', 'SDK2003SP2KEY': 'HKLM\\SOFTWARE\\Microsoft\\MicrosoftSDK\\InstalledSDKs\\D2FF9F89-8AA2-4373-8A31-C838BF4DBBE1', 'MOZ_NO_RESET_PATH': '1', 'CONTROLFILE': '"e:\\buildbot-tac.control"', 'SDKROOTKEY': 'HKLM\\SOFTWARE\\Microsoft\\MicrosoftSDK\\InstalledSDKs', 'FRAMEWORKVERSION': 'v2.0.50727', 'TMP': 'C:/DOCUME~1/cltbld/LOCALS~1/Temp', 'COMPUTERNAME': 'MW32-IX-SLAVE17', 'USERDOMAIN': 'MW32-IX-SLAVE17', 'TRY_PASSWORD': '"secret"', 'VS90COMNTOOLS': 'D:\\msvs9\\Common7\\Tools\\', 'COMMONPROGRAMFILES': 'C:\\Program Files\\Common Files', 'MOZILLABUILDPATH': '\\mozilla-build\\', 'HOME': 'c:/Documents and Settings/cltbld', 'SDKDIR': 'D:\\sdks\\v7.0\\', 'TACSCRIPT': '"d:\\tools\\buildbot-helpers\\buildbot-tac.py"', 'MSYS_TBOX_CLIENT_CVS_DIR': 'e:/builds/tinderbox/mozilla/tools/tinderbox', 'PROCESSOR_ARCHITECTURE': 'x86', 'ALLUSERSPROFILE': 'C:\\Documents and Settings\\All Users', '_': 'd:/mozilla-build/python25/scripts/buildbot', 'OSTYPE': 'msys', 'FRAMEWORKDIR': 'C:\\WINDOWS\\Microsoft.NET\\Framework', 'USERNAME': 'cltbld', 'MSVC9KEY': 'HKLM\\SOFTWARE\\Microsoft\\VisualStudio\\9.0\\Setup\\VC', 'PROMPT': '$P$G', 'MOZ_TOOLS': 'D:\\mozilla-build\\\\moztools', 'PATHEXT': '.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH', 'WIN32_REDIST_DIR': 'd:/msvs8/VC/redist/x86/Microsoft.VC80.CRT', 'WINDIR': 'C:\\WINDOWS', 'MSVCEXPROOTKEY': 'HKLM\\SOFTWARE\\Microsoft\\VCExpress', 'NUMBER_OF_PROCESSORS': '4', 'VC9DIR': 'D:\\msvs9\\VC\\', 'DEVENVDIR': 'D:\\msvs8\\Common7\\IDE', 'SDKVER': '7', 'USERPROFILE': 'C:\\Documents and Settings\\cltbld', 'VC8DIR': 'D:\\msvs8\\VC\\', 'TEMPVC9DIR': 'D:\\msvs9\\VC\\', 'SDKMINORVER': '0', 'HOSTTYPE': 'i686', 'LIBPATH': 'C:\\WINDOWS\\Microsoft.NET\\Framework\\v2.0.50727;D:\\msvs8\\VC\\ATLMFC\\LIB', 'PROCESSOR_IDENTIFIER': 'x86 Family 6 Model 30 Stepping 5, GenuineIntel', 'LOGNAME': 'cltbld', 'PROGRAMFILES': 'C:\\Program Files', 'PROCESSOR_REVISION': '1e05', 'PATH': 'D:\\mozilla-build\\msys\\local\\bin;d:\\mozilla-build\\wget;d:\\mozilla-build\\7zip;d:\\mozilla-build\\blat261\\full;d:\\mozilla-build\\python25;d:\\mozilla-build\\svn-win32-1.6.3\\bin;d:\\mozilla-build\\upx203w;d:\\mozilla-build\\emacs-22.3\\bin;d:\\mozilla-build\\info-zip;d:\\mozilla-build\\nsis-2.22;d:\\mozilla-build\\nsis-2.33u;d:\\mozilla-build\\hg;d:\\mozilla-build\\python25\\Scripts;d:\\mozilla-build\\kdiff3;d:\\mozilla-build\\vim\\vim72;.;D:\\mozilla-build\\msys\\local\\bin;D:\\mozilla-build\\msys\\mingw\\bin;D:\\mozilla-build\\msys\\bin;d:\\sdks\\v7.0\\bin;d:\\msvs8\\Common7\\IDE;d:\\msvs8\\VC\\BIN;d:\\msvs8\\Common7\\Tools;d:\\msvs8\\Common7\\Tools\\bin;d:\\msvs8\\VC\\PlatformSDK\\bin;d:\\msvs8\\SDK\\v2.0\\bin;c:\\WINDOWS\\Microsoft.NET\\Framework\\v2.0.50727;d:\\msvs8\\VC\\VCPackages;c:\\WINDOWS\\system32;c:\\WINDOWS;c:\\WINDOWS\\System32\\Wbem;c:\\Program Files\\Intel\\DMIX;d:\\mozilla-build\\python25;d:\\mercurial;c:\\Program Files\\Microsoft SQL Server\\90\\Tools\\binn\\;d:\\sdks\\tegra042\\tools;d:\\sdks\\tegra042\\platformlibs\\bin\\winxp\\x86\\release;d:\\sdks\\tegra042\\3rdparty\\bin\\winxp\\x86\\release;d:\\mozilla-build\\moztools\\bin', 'SSH_AGENT_PID': '2448', 'MAKE_MODE': 'unix', 'TERM': 'cygwin', 'MSVC6KEY': 'HKLM\\SOFTWARE\\Microsoft\\VisualStudio\\6.0\\Setup\\Microsoft Visual C++', 'WIN64': '0', 'VSINSTALLDIR': 'D:\\msvs8', '!::': '::\\', 'SESSIONNAME': 'Console', 'SDK61KEY': 'HKLM\\SOFTWARE\\Microsoft\\Microsoft SDKs\\Windows\\v6.1', 'FRAMEWORKSDKDIR': 'D:\\msvs8\\SDK\\v2.0', 'MSVC8EXPRESSKEY': 'HKLM\\SOFTWARE\\Microsoft\\VCExpress\\8.0\\Setup\\VC', 'LOG': 'c:/tmp/buildbot-sh-startup.log', 'SSH_AUTH_SOCK': 'C:/DOCUME~1/cltbld/LOCALS~1/Temp/ssh-hUumHu2396/agent.2396', 'SDK6AKEY': 'HKLM\\SOFTWARE\\Microsoft\\Microsoft SDKs\\Windows\\v6.0A', 'BOOTMODE': 'BKSTD', 'PASSWORD': '"secret"', 'FP_NO_HOST_CHECK': 'NO', 'MOZ_MSVCVERSION': '8', 'BUILDSLAVE_TACFILE': '"e:\\builds\\moz2_slave\\buildbot.tac"', 'PWD': 'c:/Documents and Settings/cltbld', 'MSVC8KEY': 'HKLM\\SOFTWARE\\Microsoft\\VisualStudio\\8.0\\Setup\\VC', 'VCINSTALLDIR': 'D:\\msvs8\\VC'}
2010/04/21 20:32 -0700 [Broker,client]   closing stdin
2010/04/21 20:32 -0700 [Broker,client]   using PTY: False
2010/04/21 20:33 -0700 [-] Received SIGBREAK, shutting down.
2010/04/21 20:33 -0700 [-] stopCommand: halting current command <buildbot.slave.commands.SlaveShellCommand instance at 0x019AE1E8>
2010/04/21 20:33 -0700 [-] command interrupted
2010/04/21 20:33 -0700 [-] trying process.signalProcess('KILL')
2010/04/21 20:33 -0700 [-]  signal KILL sent successfully
2010/04/21 20:33 -0700 [Broker,client] lost remote
2010/04/21 20:33 -0700 [Broker,client] lost remote
2010/04/21 20:33 -0700 [Broker,client] lost remote
2010/04/21 20:33 -0700 [Broker,client] lost remote
2010/04/21 20:33 -0700 [Broker,client] lost remote
2010/04/21 20:33 -0700 [Broker,client] lost remote
2010/04/21 20:33 -0700 [Broker,client] lost remote
2010/04/21 20:33 -0700 [Broker,client] lost remote
2010/04/21 20:33 -0700 [Broker,client] lost remote
2010/04/21 20:33 -0700 [Broker,client] lost remote
2010/04/21 20:33 -0700 [Broker,client] lost remote
2010/04/21 20:33 -0700 [Broker,client] lost remote
2010/04/21 20:33 -0700 [Broker,client] lost remote
2010/04/21 20:33 -0700 [Broker,client] lost remote
2010/04/21 20:33 -0700 [Broker,client] lost remote
2010/04/21 20:33 -0700 [Broker,client] lost remote
2010/04/21 20:33 -0700 [Broker,client] lost remote step
2010/04/21 20:33 -0700 [Broker,client] Stopping factory <buildbot.slave.bot.BotFactory instance at 0x0149A788>
2010/04/21 20:33 -0700 [-] Main loop terminated.
2010/04/21 20:33 -0700 [-] Server Shut Down.
2010/04/21 20:35 -0700 [-] Log opened.
2010/04/21 20:35 -0700 [-] twistd 2.5.0 (d:\mozilla-build\python25\python.exe 2.5.0) starting up
2010/04/21 20:35 -0700 [-] reactor class: <class 'twisted.internet.win32eventreactor.Win32Reactor'>
2010/04/21 20:35 -0700 [-] Loading buildbot.tac...
2010/04/21 20:35 -0700 [-] Creating BuildSlave -- buildbot.version: 0.7.10p1
2010/04/21 20:35 -0700 [-] Loaded.
2010/04/21 20:35 -0700 [-] Starting factory <buildbot.slave.bot.BotFactory instance at 0x0149A788>
2010/04/21 20:35 -0700 [Broker,client] message from master: attached
2010/04/21 20:35 -0700 [-] Log opened.
2010/04/21 20:35 -0700 [-] twistd 2.5.0 (d:\mozilla-build\python25\python.exe 2.5.0) starting up
2010/04/21 20:35 -0700 [-] reactor class: <class 'twisted.internet.win32eventreactor.Win32Reactor'>
2010/04/21 20:35 -0700 [-] Loading buildbot.tac...
2010/04/21 20:35 -0700 [-] Creating BuildSlave -- buildbot.version: 0.7.10p1
2010/04/21 20:35 -0700 [-] Loaded.
2010/04/21 20:35 -0700 [-] Starting factory <buildbot.slave.bot.BotFactory instance at 0x0149A788>
2010/04/21 20:35 -0700 [Broker,client] ReconnectingPBClientFactory.failedToGetPerspective
2010/04/21 20:35 -0700 [Broker,client] Unhandled Error
	Traceback from remote host -- Traceback (most recent call last):
	  File "/tools/twisted-8.0.1/lib/python2.5/site-packages/twisted/internet/defer.py", line 194, in addCallback
	    callbackKeywords=kw)
	  File "/tools/twisted-8.0.1/lib/python2.5/site-packages/twisted/internet/defer.py", line 185, in addCallbacks
	    self._runCallbacks()
	  File "/tools/twisted-8.0.1/lib/python2.5/site-packages/twisted/internet/defer.py", line 323, in _runCallbacks
	    self.result = callback(self.result, *args, **kw)
	  File "/tools/buildbot-production/lib/python2.5/site-packages/buildbot-0.7.10p1-py2.5.egg/buildbot/master.py", line 359, in requestAvatar
	    d = defer.maybeDeferred(p.attached, mind)
	--- <exception caught here> ---
	  File "/tools/twisted-8.0.1/lib/python2.5/site-packages/twisted/internet/defer.py", line 105, in maybeDeferred
	    result = f(*args, **kw)
	  File "/tools/buildbot-production/lib/python2.5/site-packages/buildbot-0.7.10p1-py2.5.egg/buildbot/buildslave.py", line 168, in attached
	    d = self.disconnect()
	  File "/tools/buildbot-production/lib/python2.5/site-packages/buildbot-0.7.10p1-py2.5.egg/buildbot/buildslave.py", line 269, in disconnect
	    return self._disconnect(self.slave)
	  File "/tools/buildbot-production/lib/python2.5/site-packages/buildbot-0.7.10p1-py2.5.egg/buildbot/buildslave.py", line 284, in _disconnect
	    slave.notifyOnDisconnect(_disconnected)
	  File "/tools/twisted-8.0.1/lib/python2.5/site-packages/twisted/spread/pb.py", line 297, in notifyOnDisconnect
	    self.broker.notifyOnDisconnect(self._disconnected)
	  File "/tools/twisted-8.0.1/lib/python2.5/site-packages/twisted/spread/pb.py", line 618, in notifyOnDisconnect
	    self.disconnects.append(notifier)
	exceptions.AttributeError: 'NoneType' object has no attribute 'append'

Comment 1

[Chris AtLee [:catlee]]

slave20 and 24 were also in this state

Comment 2

[[e-mail] (:bhearsum)]

20 was slightly different -- it had the same exception, but managed to reboot and still didn't connect to the master.

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Are we still seeing problems with:

mw32-ix-slave17
mw32-ix-slave20
mw32-ix-slave24

If so, have we tried reimaging and running them in staging?

Comment 4

[Mike Taylor [:bear]]

most of the code implicated in those stacktraces have been upgraded and they are not exhibiting anything like this now - will file new bugs if they do