Closed Bug 640323 Opened 15 years ago Closed 15 years ago

buildbot-master2:8011 had been down since yesterday (2011-03-08 08:15)

Categories

(Release Engineering :: General, defect, P2)

Product:
Release Engineering
Component:
General
Platform:
x86
All
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: armenzg, Assigned: armenzg)

Details

(Whiteboard: [badmaster][slaveduty])

Attachments

(2 files)

twistd.log
675.01 KB, text/plain
Details
joint log from the reconfig to the halt
1.68 MB, text/plain
Details
Attached file twistd.log —
[cltbld@buildbot-master2 tests_master5]$ ls -l /builds/buildbot/tests_master5/m aster/twistd.log -rw-rw-r-- 1 cltbld cltbld 691215 Mar 8 08:15 /builds/buildbot/tests_master5/ma ster/twistd.log 2011-03-08 08:13:39-0800 [Broker,41927,10.12.50.213] <Build Rev3 WINNT 6.1 tryserver debug test mochitests-5/5>.lostRemote 2011-03-08 08:13:39-0800 [Broker,41927,10.12.50.213] stopping currentStep <buildbotcustom.steps.misc.UnpackTest instance at 0x61ab0a6c> 2011-03-08 08:13:39-0800 [Broker,41927,10.12.50.213] addCompleteLog(interrupt) 2011-03-08 08:13:39-0800 [Broker,41927,10.12.50.213] RemoteCommand.interrupt <RemoteShellCommand '['unzip', '-o', u'firefox-4.0b13pre.en-US.win32.tests.zip', 'bin*', 'certs*', 'mochitest*']'> [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion. ] 2011-03-08 08:13:39-0800 [Broker,41927,10.12.50.213] RemoteCommand.disconnect: lost slave 2011-03-08 08:13:39-0800 [Broker,41927,10.12.50.213] releaseLocks(<buildbotcustom.steps.misc.UnpackTest instance at 0x61ab0a6c>): [] 2011-03-08 08:13:39-0800 [Broker,41927,10.12.50.213] step 'unpack mochitest tests' complete: retry 2011-03-08 08:13:39-0800 [Broker,41927,10.12.50.213] <Build Rev3 WINNT 6.1 tryserver debug test mochitests-5/5>: build finished 2011-03-08 08:13:39-0800 [Broker,41927,10.12.50.213] sending mail (24754 bytes) to ['tinderbox-daemon@tinderbox.mozilla.org'] 2011-03-08 08:13:39-0800 [Broker,41927,10.12.50.213] Starting factory <twisted.mail.smtp.ESMTPSenderFactory instance at 0x2e330d0c> 2011-03-08 08:13:44-0800 [Broker,41927,10.12.50.213] loop triggered while service disabled; ignoring trigger 2011-03-08 08:13:44-0800 [Broker,41927,10.12.50.213] expiring old connection 2011-03-08 08:13:44-0800 [Broker,41927,10.12.50.213] adbapi closing: MySQLdb 2011-03-08 08:13:44-0800 [Broker,41927,10.12.50.213] adbapi connecting: MySQLdb {'use_unicode': True, 'passwd': 'H63732fW54vm', 'charset': 'utf8', 'db': 'buildbot_schedulers', 'host': 'tm-b01-master01.mozilla.org', 'user': 'buildbot'} 2011-03-08 08:13:44-0800 [Broker,41927,10.12.50.213] loop triggered while service disabled; ignoring trigger 2011-03-08 08:13:44-0800 [Broker,41902,10.12.49.251] BuildSlave.detached(talos-r3-leopard-016) 2011-03-08 08:13:44-0800 [Broker,41902,10.12.49.251] <Build Rev3 MacOSX Leopard 10.5.8 tryserver debug test mochitests-4/5>.lostRemote 2011-03-08 08:13:44-0800 [Broker,41902,10.12.49.251] stopping currentStep <buildbotcustom.steps.unittest.MozillaPackagedMochitests object at 0x3e2e5a6c> 2011-03-08 08:13:44-0800 [Broker,41902,10.12.49.251] addCompleteLog(interrupt) 2011-03-08 08:13:44-0800 [Broker,41902,10.12.49.251] RemoteCommand.interrupt <RemoteShellCommand '['python', 'mochitest/runtests.py', '--appname=./MinefieldDebug.app/Contents/MacOS/firefox-bin', '--utility-path=bin', '--extra-profile-file=bin/plugins', '--certificate-path=certs', '--autorun', '--close-when-done', '--console-level=INFO', '--symbols-path=symbols', '--total-chunks', '5', '--this-chunk', '4', '--chunk-by-dir', '4']'> [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion. ] 2011-03-08 08:13:44-0800 [Broker,41902,10.12.49.251] RemoteCommand.disconnect: lost slave 2011-03-08 08:13:44-0800 [Broker,41902,10.12.49.251] Pulse <buildbotcustom.status.pulse.PulseStatus instance at 0x1e6bf84c>: Initiating push of 5 events 2011-03-08 08:13:44-0800 [Broker,41902,10.12.49.251] releaseLocks(<buildbotcustom.steps.unittest.MozillaPackagedMochitests object at 0x3e2e5a6c>): [] 2011-03-08 08:13:44-0800 [Broker,41902,10.12.49.251] step 'mochitest-plain-4' complete: retry 2011-03-08 08:13:44-0800 [Broker,41902,10.12.49.251] <Build Rev3 MacOSX Leopard 10.5.8 tryserver debug test mochitests-4/5>: build finished 2011-03-08 08:13:45-0800 [Broker,41902,10.12.49.251] Pulse <buildbotcustom.status.pulse.PulseStatus instance at 0x1e6bf84c>: Sent 5 events in 0.52 seconds 2011-03-08 08:13:45-0800 [Broker,41902,10.12.49.251] sending mail (278214 bytes) to ['tinderbox-daemon@tinderbox.mozilla.org'] 2011-03-08 08:13:45-0800 [Broker,41902,10.12.49.251] Starting factory <twisted.mail.smtp.ESMTPSenderFactory instance at 0x109bab4c> 2011-03-08 08:13:49-0800 [Broker,41902,10.12.49.251] loop triggered while service disabled; ignoring trigger 2011-03-08 08:13:50-0800 [Broker,41902,10.12.49.251] expiring old connection 2011-03-08 08:13:50-0800 [Broker,41902,10.12.49.251] adbapi closing: MySQLdb 2011-03-08 08:13:50-0800 [Broker,41902,10.12.49.251] adbapi connecting: MySQLdb {'use_unicode': True, 'passwd': 'H63732fW54vm', 'charset': 'utf8', 'db': 'buildbot_schedulers', 'host': 'tm-b01-master01.mozilla.org', 'user': 'buildbot'} 2011-03-08 08:13:50-0800 [Broker,41902,10.12.49.251] loop triggered while service disabled; ignoring trigger 2011-03-08 08:13:50-0800 [Broker,41930,10.12.49.147] BuildSlave.detached(talos-r3-fed-020) 2011-03-08 08:13:50-0800 [Broker,41926,10.12.49.210] BuildSlave.detached(talos-r3-fed64-030) 2011-03-08 08:13:50-0800 [Broker,37232,10.12.50.157] BuildSlave.detached(talos-r3-xp-049) 2011-03-08 08:13:50-0800 [Broker,37232,10.12.50.157] <Build Rev3 WINNT 5.1 jaegermonkey opt test xpcshell>.lostRemote 2011-03-08 08:13:50-0800 [Broker,37232,10.12.50.157] stopping currentStep <buildbotcustom.steps.unittest.MozillaPackagedXPCShellTests instance at 0x5621bdcc> 2011-03-08 08:13:50-0800 [Broker,37232,10.12.50.157] addCompleteLog(interrupt) 2011-03-08 08:13:50-0800 [Broker,37232,10.12.50.157] RemoteCommand.interrupt <RemoteShellCommand '['bash', '-c', 'if [ ! -d firefox/plugins ]; then mkdir firefox/plugins; fi && cp bin/xpcshell.exe firefox && cp -R bin/components/* firefox/components/ && cp -R bin/plugins/* firefox/plugins/ && python -u xpcshell/runxpcshelltests.py --symbols-path=symbols --manifest=xpcshell/tests/all-test-dirs.list firefox/xpcshell.exe']'> [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion. ] 2011-03-08 08:13:50-0800 [Broker,37232,10.12.50.157] RemoteCommand.disconnect: lost slave 2011-03-08 08:13:50-0800 [Broker,37232,10.12.50.157] BuildStep.failed, traceback follows 2011-03-08 08:13:50-0800 [Broker,37232,10.12.50.157] Unhandled Error Traceback (most recent call last): File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/twisted/internet/defer.py", line 249, in addCallbacks self._runCallbacks() File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks self.result = callback(self.result, *args, **kw) File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/twisted/internet/defer.py", line 318, in callback self._startRunCallbacks(result) File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/twisted/internet/defer.py", line 424, in _startRunCallbacks self._runCallbacks() --- <exception caught here> --- File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks self.result = callback(self.result, *args, **kw) File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/buildstep.py", line 1118, in checkDisconnect return self.finished(RETRY) File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/buildstep.py", line 867, in finished self.step_status.stepFinished(results) File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/status/builder.py", line 1047, in stepFinished if os.path.getsize(loog.getFilename()) > logCompressionLimit: File "/builds/buildbot/tests_master5/lib/python2.6/genericpath.py", line 49, in getsize return os.stat(filename).st_size exceptions.OSError: [Errno 2] No such file or directory: './jaegermonkey_xp_test-xpcshell/1-log-xpcshell-stdio' 2011-03-08 08:13:50-0800 [Broker,37232,10.12.50.157] exception during failure processing 2011-03-08 08:13:50-0800 [Broker,37232,10.12.50.157] Unhandled Error Traceback (most recent call last): File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks self.result = callback(self.result, *args, **kw) File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/twisted/internet/defer.py", line 318, in callback self._startRunCallbacks(result) File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/twisted/internet/defer.py", line 424, in _startRunCallbacks self._runCallbacks() File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks self.result = callback(self.result, *args, **kw) --- <exception caught here> --- File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/buildstep.py", line 879, in failed self.progress.finish() File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/status/progress.py", line 70, in finish self.buildProgress.stepFinished(self.name) File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/status/progress.py", line 149, in stepFinished assert(stepname not in self.finishedSteps) exceptions.AssertionError: 2011-03-08 08:13:50-0800 [Broker,37232,10.12.50.157] releaseLocks(<buildbotcustom.steps.unittest.MozillaPackagedXPCShellTests instance at 0x5621bdcc>): [] 2011-03-08 08:13:50-0800 [Broker,37232,10.12.50.157] BuildStep.failed now firing callback 2011-03-08 08:13:50-0800 [Broker,37232,10.12.50.157] step 'xpcshell' complete: exception 2011-03-08 08:13:50-0800 [Broker,37232,10.12.50.157] <Build Rev3 WINNT 5.1 jaegermonkey opt test xpcshell>: build finished 2011-03-08 08:13:50-0800 [Broker,37232,10.12.50.157] Exception caught notifying <buildbot.status.tinderbox.TinderboxMailNotifier instance at 0x3ed4faac> of buildFinished event 2011-03-08 08:13:50-0800 [Broker,37232,10.12.50.157] Unhandled Error Traceback (most recent call last): File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/status/builder.py", line 1388, in buildFinished w.callback(self) File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/twisted/internet/defer.py", line 318, in callback self._startRunCallbacks(result) File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/twisted/internet/defer.py", line 424, in _startRunCallbacks self._runCallbacks() File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks self.result = callback(self.result, *args, **kw) --- <exception caught here> --- File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/status/builder.py", line 2044, in _buildFinished w.buildFinished(name, s, results) File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/status/mail.py", line 360, in buildFinished return self.buildMessage(name, build, results) File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/status/tinderbox.py", line 226, in buildMessage logText = log.getTextWithHeaders() File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/status/builder.py", line 324, in getTextWithHeaders return "".join(self.getChunks(onlyText=True)) File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/status/builder.py", line 339, in getChunks f = self.getFile() File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/status/builder.py", line 317, in getFile return open(self.getFilename(), "r") exceptions.IOError: [Errno 2] No such file or directory: './jaegermonkey_xp_test-xpcshell/1-log-get_buildername-output' 2011-03-08 08:13:55-0800 [Broker,37232,10.12.50.157] loop triggered while service disabled; ignoring trigger 2011-03-08 08:13:55-0800 [Broker,37232,10.12.50.157] loop triggered while service disabled; ignoring trigger 2011-03-08 08:13:55-0800 [Broker,41932,10.12.50.94] BuildSlave.detached(talos-r3-snow-041) 2011-03-08 08:13:55-0800 [Broker,41928,10.12.49.155] BuildSlave.detached(talos-r3-fed-028) 2011-03-08 08:13:55-0800 [Broker,41873,10.12.49.151] BuildSlave.detached(talos-r3-fed-024) 2011-03-08 08:13:55-0800 [Broker,41875,10.12.49.145] BuildSlave.detached(talos-r3-fed-018) 2011-03-08 08:13:55-0800 [Broker,41865,10.12.49.197] BuildSlave.detached(talos-r3-fed64-017) 2011-03-08 08:13:55-0800 [Broker,41916,10.12.50.91] BuildSlave.detached(talos-r3-snow-038) 2011-03-08 08:13:55-0800 [Broker,41916,10.12.50.91] <Build Rev3 MacOSX Snow Leopard 10.6.2 build-system talos svg>.lostRemote 2011-03-08 08:13:55-0800 [Broker,41916,10.12.50.91] stopping currentStep <buildbotcustom.steps.talos.MozillaRunPerfTests instance at 0x575bbd8c> 2011-03-08 08:13:55-0800 [Broker,41916,10.12.50.91] addCompleteLog(interrupt) 2011-03-08 08:13:55-0800 [Broker,41916,10.12.50.91] RemoteCommand.interrupt <RemoteShellCommand '['python', 'run_tests.py', '--noisy', '20110308_0806_config.yml']'> [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion. ] 2011-03-08 08:13:55-0800 [Broker,41916,10.12.50.91] RemoteCommand.disconnect: lost slave 2011-03-08 08:13:55-0800 [Broker,41916,10.12.50.91] releaseLocks(<buildbotcustom.steps.talos.MozillaRunPerfTests instance at 0x575bbd8c>): [] 2011-03-08 08:13:55-0800 [Broker,41916,10.12.50.91] step 'Run performance tests' complete: retry 2011-03-08 08:13:55-0800 [Broker,41916,10.12.50.91] <Build Rev3 MacOSX Snow Leopard 10.6.2 build-system talos svg>: build finished 2011-03-08 08:13:55-0800 [Broker,41916,10.12.50.91] sending mail (45259 bytes) to ['tinderbox-daemon@tinderbox.mozilla.org'] 2011-03-08 08:13:55-0800 [Broker,41916,10.12.50.91] Starting factory <twisted.mail.smtp.ESMTPSenderFactory instance at 0x35244d2c> 2011-03-08 08:14:00-0800 [Broker,41916,10.12.50.91] loop triggered while service disabled; ignoring trigger 2011-03-08 08:14:00-0800 [Broker,41916,10.12.50.91] loop triggered while service disabled; ignoring trigger 2011-03-08 08:14:00-0800 [Broker,41920,10.12.49.188] BuildSlave.detached(talos-r3-fed64-008) 2011-03-08 08:14:00-0800 [Broker,41862,10.12.49.234] BuildSlave.detached(talos-r3-fed64-054) 2011-03-08 08:14:00-0800 [Broker,41905,10.12.49.167] BuildSlave.detached(talos-r3-fed-040) 2011-03-08 08:14:00-0800 [Broker,41922,10.12.50.100] BuildSlave.detached(talos-r3-snow-047) 2011-03-08 08:14:00-0800 [Broker,41922,10.12.50.100] <Build Rev3 MacOSX Snow Leopard 10.6.2 build-system opt test xpcshell>.lostRemote 2011-03-08 08:14:00-0800 [Broker,41922,10.12.50.100] stopping currentStep <buildbotcustom.steps.unittest.MozillaPackagedXPCShellTests instance at 0x593cb54c> 2011-03-08 08:14:00-0800 [Broker,41922,10.12.50.100] addCompleteLog(interrupt) 2011-03-08 08:14:00-0800 [Broker,41922,10.12.50.100] RemoteCommand.interrupt <RemoteShellCommand '['bash', '-c', 'if [ ! -d ./Minefield.app/Contents/MacOS/plugins ]; then mkdir ./Minefield.app/Contents/MacOS/plugins; fi && cp bin/xpcshell ./Minefield.app/Contents/MacOS && cp -R bin/components/* ./Minefield.app/Contents/MacOS/components/ && cp -R bin/plugins/* ./Minefield.app/Contents/MacOS/plugins/ && python -u xpcshell/runxpcshelltests.py --symbols-path=symbols --manifest=xpcshell/tests/all-test-dirs.list ./Minefield.app/Contents/MacOS/xpcshell']'> [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion. ] 2011-03-08 08:14:00-0800 [Broker,41922,10.12.50.100] RemoteCommand.disconnect: lost slave 2011-03-08 08:14:00-0800 [Broker,41922,10.12.50.100] releaseLocks(<buildbotcustom.steps.unittest.MozillaPackagedXPCShellTests instance at 0x593cb54c>): [] 2011-03-08 08:14:00-0800 [Broker,41922,10.12.50.100] step 'xpcshell' complete: retry 2011-03-08 08:14:00-0800 [Broker,41922,10.12.50.100] <Build Rev3 MacOSX Snow Leopard 10.6.2 build-system opt test xpcshell>: build finished 2011-03-08 08:14:00-0800 [Broker,41922,10.12.50.100] sending mail (47763 bytes) to ['tinderbox-daemon@tinderbox.mozilla.org'] 2011-03-08 08:14:00-0800 [Broker,41922,10.12.50.100] Starting factory <twisted.mail.smtp.ESMTPSenderFactory instance at 0x4df6d06c> 2011-03-08 08:14:05-0800 [Broker,41922,10.12.50.100] loop triggered while service disabled; ignoring trigger 2011-03-08 08:14:05-0800 [Broker,41922,10.12.50.100] loop triggered while service disabled; ignoring trigger 2011-03-08 08:14:05-0800 [Broker,40833,10.12.50.227] BuildSlave.detached(t-r3-w764-014) 2011-03-08 08:14:05-0800 [Broker,41910,10.12.50.83] BuildSlave.detached(talos-r3-snow-030) 2011-03-08 08:14:05-0800 [Broker,41910,10.12.50.83] <Build Rev3 MacOSX Snow Leopard 10.6.2 tryserver debug test mochitests-1/5>.lostRemote 2011-03-08 08:14:05-0800 [Broker,41910,10.12.50.83] stopping currentStep <buildbotcustom.steps.unittest.MozillaPackagedMochitests object at 0x14abebac> 2011-03-08 08:14:05-0800 [Broker,41910,10.12.50.83] addCompleteLog(interrupt) 2011-03-08 08:14:06-0800 [Broker,41910,10.12.50.83] RemoteCommand.interrupt <RemoteShellCommand '['python', 'mochitest/runtests.py', '--appname=./MinefieldDebug.app/Contents/MacOS/firefox-bin', '--utility-path=bin', '--extra-profile-file=bin/plugins', '--certificate-path=certs', '--autorun', '--close-when-done', '--console-level=INFO', '--symbols-path=symbols', '--total-chunks', '5', '--this-chunk', '1', '--chunk-by-dir', '4']'> [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion. ] 2011-03-08 08:14:06-0800 [Broker,41910,10.12.50.83] RemoteCommand.disconnect: lost slave 2011-03-08 08:14:06-0800 [Broker,41910,10.12.50.83] releaseLocks(<buildbotcustom.steps.unittest.MozillaPackagedMochitests object at 0x14abebac>): [] 2011-03-08 08:14:06-0800 [Broker,41910,10.12.50.83] step 'mochitest-plain-1' complete: retry 2011-03-08 08:14:06-0800 [Broker,41910,10.12.50.83] <Build Rev3 MacOSX Snow Leopard 10.6.2 tryserver debug test mochitests-1/5>: build finished 2011-03-08 08:14:12-0800 [Broker,41910,10.12.50.83] sending mail (1098424 bytes) to ['tinderbox-daemon@tinderbox.mozilla.org'] 2011-03-08 08:14:12-0800 [Broker,41910,10.12.50.83] Starting factory <twisted.mail.smtp.ESMTPSenderFactory instance at 0x1a63750c> 2011-03-08 08:14:16-0800 [Broker,41910,10.12.50.83] loop triggered while service disabled; ignoring trigger 2011-03-08 08:14:16-0800 [Broker,41910,10.12.50.83] loop triggered while service disabled; ignoring trigger 2011-03-08 08:14:16-0800 [Broker,41925,10.12.49.162] BuildSlave.detached(talos-r3-fed-035) 2011-03-08 08:14:16-0800 [Broker,41887,10.12.49.176] BuildSlave.detached(talos-r3-fed-049) 2011-03-08 08:14:16-0800 [Broker,41914,10.12.50.101] BuildSlave.detached(talos-r3-snow-048) 2011-03-08 08:14:17-0800 [Broker,41892,10.12.50.103] BuildSlave.detached(talos-r3-snow-050) 2011-03-08 08:14:17-0800 [Broker,41892,10.12.50.103] <Build Rev3 MacOSX Snow Leopard 10.6.2 tryserver debug test mochitests-4/5>.lostRemote 2011-03-08 08:14:17-0800 [Broker,41892,10.12.50.103] stopping currentStep <buildbotcustom.steps.unittest.MozillaPackagedMochitests object at 0x3766bd2c> 2011-03-08 08:14:17-0800 [Broker,41892,10.12.50.103] addCompleteLog(interrupt) 2011-03-08 08:14:17-0800 [Broker,41892,10.12.50.103] RemoteCommand.interrupt <RemoteShellCommand '['python', 'mochitest/runtests.py', '--appname=./MinefieldDebug.app/Contents/MacOS/firefox-bin', '--utility-path=bin', '--extra-profile-file=bin/plugins', '--certificate-path=certs', '--autorun', '--close-when-done', '--console-level=INFO', '--symbols-path=symbols', '--total-chunks', '5', '--this-chunk', '4', '--chunk-by-dir', '4']'> [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion. ] 2011-03-08 08:14:17-0800 [Broker,41892,10.12.50.103] RemoteCommand.disconnect: lost slave 2011-03-08 08:14:17-0800 [Broker,41892,10.12.50.103] releaseLocks(<buildbotcustom.steps.unittest.MozillaPackagedMochitests object at 0x3766bd2c>): [] 2011-03-08 08:14:17-0800 [Broker,41892,10.12.50.103] step 'mochitest-plain-4' complete: retry 2011-03-08 08:14:17-0800 [Broker,41892,10.12.50.103] <Build Rev3 MacOSX Snow Leopard 10.6.2 tryserver debug test mochitests-4/5>: build finished 2011-03-08 08:14:20-0800 [Broker,41892,10.12.50.103] sending mail (1719080 bytes) to ['tinderbox-daemon@tinderbox.mozilla.org'] 2011-03-08 08:14:20-0800 [Broker,41892,10.12.50.103] Starting factory <twisted.mail.smtp.ESMTPSenderFactory instance at 0x2e18e6ec> 2011-03-08 08:14:24-0800 [Broker,41892,10.12.50.103] loop triggered while service disabled; ignoring trigger 2011-03-08 08:14:24-0800 [Broker,41892,10.12.50.103] loop triggered while service disabled; ignoring trigger 2011-03-08 08:14:24-0800 [Broker,41924,10.12.50.193] BuildStep.failed, traceback follows 2011-03-08 08:14:24-0800 [Broker,41924,10.12.50.193] Unhandled Error Traceback (most recent call last): Failure: twisted.spread.pb.PBConnectionLost: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion: Connection lost. ] 2011-03-08 08:14:24-0800 [Broker,41924,10.12.50.193] addHTMLLog(err.html) 2011-03-08 08:14:24-0800 [Broker,41924,10.12.50.193] addCompleteLog(err.text) 2011-03-08 08:14:25-0800 [Broker,41924,10.12.50.193] releaseLocks(<buildbotcustom.steps.base.C instance at 0x6a438bec>): [] 2011-03-08 08:14:25-0800 [Broker,41924,10.12.50.193] BuildStep.failed now firing callback 2011-03-08 08:14:25-0800 [Broker,41924,10.12.50.193] step 'get build info' complete: exception 2011-03-08 08:14:25-0800 [Broker,41924,10.12.50.193] ShellCommand.startCommand(cmd=<RemoteShellCommand '['python', 'tools/buildfarm/maintenance/count_and_reboot.py', '-f', '../reboot_count.txt', '-n', '1', '-z']'>) 2011-03-08 08:14:25-0800 [Broker,41924,10.12.50.193] cmd.args = {'workdir': '.', 'timeout': 1200, 'env': {'MOZ_CRASHREPORTER_NO_REPORT': '1', 'XPCOM_DEBUG_BREAK': 'warn', 'NO_EM_RESTART': '1', 'MOZ_NO_REMOTE': '1', 'PATH': 'C:\\mozilla-build;C:\\mozilla-build\\msys\\bin;C:\\mozilla-build\\msys\\local\\bin;C:\\mozilla-build\\Python25;C:\\mozilla-build\\Python25\\Scripts;C:\\mozilla-build\\hg;C:\\mozilla-build\\7zip;C:\\mozilla-build\\upx203w;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32;C:\\WINDOWS;'}, 'want_stdout': 1, 'usePTY': 'slave-config', 'maxTime': None, 'logEnviron': True, 'want_stderr': 1, 'logfiles': {}} 2011-03-08 08:14:25-0800 [Broker,41924,10.12.50.193] <RemoteShellCommand '['python', 'tools/buildfarm/maintenance/count_and_reboot.py', '-f', '../reboot_count.txt', '-n', '1', '-z']'>: RemoteCommand.run [774996] 2011-03-08 08:14:25-0800 [Broker,41924,10.12.50.193] command '['python', 'tools/buildfarm/maintenance/count_and_reboot.py', '-f', '../reboot_count.txt', '-n', '1', '-z']' in dir '.' 2011-03-08 08:14:25-0800 [Broker,41924,10.12.50.193] LoggedRemoteCommand.start 2011-03-08 08:14:25-0800 [Broker,41924,10.12.50.193] BuildStep.failed, traceback follows 2011-03-08 08:14:25-0800 [Broker,41924,10.12.50.193] Unhandled Error Traceback (most recent call last): File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/steps/shell.py", line 225, in start self.startCommand(cmd, warnings) File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/buildstep.py", line 1062, in startCommand d = self.runCommand(cmd) # might raise ConnectionLost File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/buildstep.py", line 993, in runCommand d = c.run(self, self.remote) File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/buildstep.py", line 77, in run d = defer.maybeDeferred(self.start) --- <exception caught here> --- File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/twisted/internet/defer.py", line 125, in maybeDeferred result = f(*args, **kw) File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/buildstep.py", line 529, in start return LoggedRemoteCommand.start(self) File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/buildstep.py", line 311, in start return RemoteCommand.start(self) File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/buildstep.py", line 112, in start self.remote_command, cmd_args) File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/twisted/spread/pb.py", line 328, in callRemote _name, args, kw) File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/twisted/spread/pb.py", line 807, in _sendMessage raise DeadReferenceError("Calling Stale Broker") twisted.spread.pb.DeadReferenceError: Calling Stale Broker 2011-03-08 08:14:25-0800 [Broker,41924,10.12.50.193] addHTMLLog(err.html) 2011-03-08 08:14:25-0800 [Broker,41924,10.12.50.193] addCompleteLog(err.text) 2011-03-08 08:14:25-0800 [Broker,41924,10.12.50.193] releaseLocks(<buildbotcustom.steps.misc.DisconnectStep instance at 0x3e941acc>): [] 2011-03-08 08:14:25-0800 [Broker,41924,10.12.50.193] BuildStep.failed now firing callback 2011-03-08 08:14:25-0800 [Broker,41924,10.12.50.193] step 'maybe_rebooting' complete: exception 2011-03-08 08:14:25-0800 [Broker,41924,10.12.50.193] <Build Rev3 WINNT 6.1 tryserver debug test mochitests-1/5>: build finished 2011-03-08 08:14:25-0800 [Broker,41924,10.12.50.193] sending mail (15073 bytes) to ['tinderbox-daemon@tinderbox.mozilla.org'] 2011-03-08 08:14:25-0800 [Broker,41924,10.12.50.193] Starting factory <twisted.mail.smtp.ESMTPSenderFactory instance at 0x5c4a0a0c> 2011-03-08 08:14:30-0800 [Broker,41924,10.12.50.193] loop triggered while service disabled; ignoring trigger 2011-03-08 08:14:30-0800 [Broker,41924,10.12.50.193] releaseLocks(<BuildSlave 'talos-r3-w7-032', current builders: Rev3 WINNT 6.1 build-system talos nochrome,Rev3 WINNT 6.1 build-system opt test jsreftest,Rev3 WINNT 6.1 tryserver talos svg,Rev3 WINNT 6.1 birch talos a11y,Rev3 WINNT 6.1 mozilla-2.0 debug test mochitest-other,Rev3 WINNT 6.1 mozilla-1.9.1 talos svg,Rev3 WINNT 6.1 mozilla-central talos dromaeo,Rev3 WINNT 6.1 jaegermonkey debug test mochitests-1/5,Rev3 WINNT 6.1 jaegermonkey debug test mochitests-3/5,Rev3 WINNT 6.1 maple debug test mochitests-1/5,Rev3 WINNT 6.1 maple debug test mochitests-3/5,Rev3 WINNT 6.1 maple debug test crashtest,Rev3 WINNT 6.1 build-system opt test xpcshell,Rev3 WINNT 6.1 services-central debug test mochitests-5/5,Rev3 WINNT 6.1 tracemonkey debug test crashtest,Rev3 WINNT 6.1 shadow-central debug test mochitest-other,Rev3 WINNT 6.1 services-central debug test mochitests-1/5,Rev3 WINNT 6.1 services-central talos dromaeo,Rev3 WINNT 6.1 maple debug test mochitests-4/5,Rev3 WINNT 6.1 mozilla-1.9.2 talos nochrome,Rev3 WINNT 6.1 release-mozilla-central opt test mochitests-5/5,Rev3 WINNT 6.1 mozilla-1.9.2 talos chrome,Rev3 WINNT 6.1 tryserver opt test mochitest-other,Rev3 WINNT 6.1 tracemonkey talos dirty,Rev3 WINNT 6.1 electrolysis debug test mochitests-1/5,Rev3 WINNT 6.1 services-central opt test jsreftest,Rev3 WINNT 6.1 build-system talos svg,Rev3 WINNT 6.1 services-central talos chrome,Rev3 WINNT 6.1 shadow-central debug test jsreftest,Rev3 WINNT 6.1 mozilla-1.9.1 talos dirty,Rev3 WINNT 6.1 birch debug test mochitests-3/5,Rev3 WINNT 6.1 cedar opt test mochitests-4/5,Rev3 WINNT 6.1 mozilla-central opt test mochitests-3/5,Rev3 WINNT 6.1 cedar opt test mochitests-2/5,Rev3 WINNT 6.1 birch opt test mochitests-5/5,Rev3 WINNT 6.1 mozilla-central opt test mochitests-2/5,Rev3 WINNT 6.1 services-central debug test mochitest-other,Rev3 WINNT 6.1 release-mozilla-central talos svg,Rev3 WINNT 6.1 electrolysis debug test jsreftest,Rev3 WINNT 6.1 tracemonkey opt test crashtest,Rev3 WINNT 6.1 mozilla-central debug test mochitests-3/5,Rev3 WINNT 6.1 shadow-central opt test mochitests-5/5,Rev3 WINNT 6.1 tryserver debug test jsreftest,Rev3 WINNT 6.1 shadow-central opt test mochitests-3/5,Rev3 WINNT 6.1 build-system debug test mochitests-5/5,Rev3 WINNT 6.1 build-system debug test mochitests-3/5,Rev3 WINNT 6.1 electrolysis opt test mochitests-1/5,Rev3 WINNT 6.1 tracemonkey debug test mochitests-5/5,Rev3 WINNT 6.1 mozilla-1.9.2 talos svg,Rev3 WINNT 6.1 tracemonkey talos a11y,Rev3 WINNT 6.1 cedar opt test crashtest,Rev3 WINNT 6.1 tracemonkey opt test mochitests-5/5,Rev3 WINNT 6.1 electrolysis opt test mochitests-4/5,Rev3 WINNT 6.1 release-mozilla-central debug test jsreftest,Rev3 WINNT 6.1 shadow-central opt test xpcshell,Rev3 WINNT 6.1 mozilla-2.0 opt test mochitests-5/5,Rev3 WINNT 6.1 tracemonkey opt test mochitests-1/5,Rev3 WINNT 6.1 shadow-central opt test jsreftest,Rev3 WINNT 6.1 places debug test mochitests-2/5,Rev3 WINNT 6.1 maple talos scroll,Rev3 WINNT 6.1 tryserver opt test mochitests-5/5,Rev3 WINNT 6.1 mozilla-2.0 opt test xpcshell,Rev3 WINNT 6.1 build-system opt test mochitests-5/5,Rev3 WINNT 6.1 cedar talos a11y,Rev3 WINNT 6.1 electrolysis debug test xpcshell,Rev3 WINNT 6.1 services-central debug test reftest,Rev3 WINNT 6.1 release-mozilla-central talos tp4,Rev3 WINNT 6.1 mozilla-central talos dirty,Rev3 WINNT 6.1 electrolysis opt test reftest,Rev3 WINNT 6.1 birch talos dromaeo,Rev3 WINNT 6.1 services-central opt test mochitests-3/5,Rev3 WINNT 6.1 release-mozilla-central debug test mochitests-4/5,Rev3 WINNT 6.1 services-central opt test mochitests-1/5,Rev3 WINNT 6.1 build-system talos a11y,Rev3 WINNT 6.1 mozilla-central talos tp4,Rev3 WINNT 6.1 services-central opt test mochitest-other,Rev3 WINNT 6.1 build-system talos dirty,Rev3 WINNT 6.1 places debug test xpcshell,Rev3 WINNT 6.1 build-system debug test crashtest,Rev3 WINNT 6.1 build-system debug test jsreftest,Rev3 WINNT 6.1 services-central talos svg,Rev3 WINNT 6.1 tracemonkey talos svg,Rev3 WINNT 6.1 tryserver debug test mochitests-4/5,Rev3 WINNT 6.1 places opt test crashtest,Rev3 WINNT 6.1 jaegermonkey opt test mochitests-5/5,Rev3 WINNT 6.1 build-system talos tp4,Rev3 WINNT 6.1 release-mozilla-1.9.2 talos chrome,Rev3 WINNT 6.1 places talos dromaeo,Rev3 WINNT 6.1 electrolysis opt test mochitests-3/5,Rev3 WINNT 6.1 places opt test reftest,Rev3 WINNT 6.1 mozilla-2.0 debug test xpcshell,Rev3 WINNT 6.1 birch talos nochrome,Rev3 WINNT 6.1 tracemonkey talos tp4,Rev3 WINNT 6.1 services-central opt test crashtest,Rev3 WINNT 6.1 places talos chrome,Rev3 WINNT 6.1 mozilla-2.0 debug test reftest,Rev3 WINNT 6.1 release-mozilla-1.9.2 talos dirty,Rev3 WINNT 6.1 cedar debug test mochitests-3/5,Rev3 WINNT 6.1 places opt test mochitests-5/5,Rev3 WINNT 6.1 cedar debug test mochitests-5/5,Rev3 WINNT 6.1 places opt test mochitests-1/5,Rev3 WINNT 6.1 cedar debug test mochitests-1/5,Rev3 WINNT 6.1 places opt test mochitests-3/5,Rev3 WINNT 6.1 mozilla-central talos chrome,Rev3 WINNT 6.1 jaegermonkey debug test mochitests-4/5,Rev3 WINNT 6.1 build-system opt test mochitests-1/5,Rev3 WINNT 6.1 release-mozilla-central talos nochrome,Rev3 WINNT 6.1 build-system opt test mochitests-3/5,Rev3 WINNT 6.1 mozilla-central opt test reftest,Rev3 WINNT 6.1 places talos nochrome,Rev3 WINNT 6.1 places opt test xpcshell,Rev3 WINNT 6.1 services-central debug test mochitests-2/5,Rev3 WINNT 6.1 maple debug test mochitests-5/5,Rev3 WINNT 6.1 maple talos dromaeo,Rev3 WINNT 6.1 birch talos dirty,Rev3 WINNT 6.1 maple opt test jsreftest,Rev3 WINNT 6.1 birch opt test mochitest-other,Rev3 WINNT 6.1 mozilla-central opt test mochitests-1/5,Rev3 WINNT 6.1 build-system opt test crashtest,Rev3 WINNT 6.1 birch opt test reftest,Rev3 WINNT 6.1 places opt test jsreftest,Rev3 WINNT 6.1 maple opt test mochitests-2/5,Rev3 WINNT 6.1 release-mozilla-central opt test mochitests-1/5,Rev3 WINNT 6.1 tracemonkey talos dromaeo,Rev3 WINNT 6.1 mozilla-2.0 opt test crashtest,Rev3 WINNT 6.1 maple debug test reftest,Rev3 WINNT 6.1 maple opt test mochitests-4/5,Rev3 WINNT 6.1 release-mozilla-central talos chrome,Rev3 WINNT 6.1 release-mozilla-central opt test mochitests-3/5,Rev3 WINNT 6.1 electrolysis debug test mochitests-3/5,Rev3 WINNT 6.1 electrolysis debug test mochitests-5/5,Rev3 WINNT 6.1 electrolysis opt test jsreftest,Rev3 WINNT 6.1 places talos dirty,Rev3 WINNT 6.1 release-mozilla-central talos dirty,Rev3 WINNT 6.1 birch opt test mochitests-3/5,Rev3 WINNT 6.1 mozilla-central debug test mochitests-2/5,Rev3 WINNT 6.1 birch opt test mochitests-1/5,Rev3 WINNT 6.1 electrolysis opt test xpcshell,Rev3 WINNT 6.1 mozilla-2.0 talos nochrome,Rev3 WINNT 6.1 release-mozilla-central debug test reftest,Rev3 WINNT 6.1 mozilla-2.0 debug test mochitests-1/5,Rev3 WINNT 6.1 mozilla-2.0 debug test mochitests-3/5,Rev3 WINNT 6.1 birch debug test mochitests-4/5,Rev3 WINNT 6.1 birch talos svg,Rev3 WINNT 6.1 tryserver opt test reftest-no-accel,Rev3 WINNT 6.1 release-mozilla-central talos a11y,Rev3 WINNT 6.1 electrolysis opt test mochitests-5/5,Rev3 WINNT 6.1 shadow-central opt test mochitests-1/5,Rev3 WINNT 6.1 birch debug test xpcshell,Rev3 WINNT 6.1 tracemonkey opt test mochitests-2/5,Rev3 WINNT 6.1 tracemonkey opt test mochitest-other,Rev3 WINNT 6.1 birch talos chrome,Rev3 WINNT 6.1 tryserver opt test mochitests-2/5,Rev3 WINNT 6.1 mozilla-1.9.1 talos dromaeo,Rev3 WINNT 6.1 maple talos nochrome,Rev3 WINNT 6.1 birch debug test jsreftest,Rev3 WINNT 6.1 places debug test jsreftest,Rev3 WINNT 6.1 shadow-central debug test mochitests-3/5,Rev3 WINNT 6.1 shadow-central debug test mochitests-5/5,Rev3 WINNT 6.1 jaegermonkey debug test xpcshell,Rev3 WINNT 6.1 services-central talos scroll,Rev3 WINNT 6.1 tryserver debug test xpcshell,Rev3 WINNT 6.1 services-central opt test mochitests-5/5,Rev3 WINNT 6.1 addontester talos addon,Rev3 WINNT 6.1 mozilla-central opt test crashtest,Rev3 WINNT 6.1 services-central debug test jsreftest,Rev3 WINNT 6.1 mozilla-1.9.2 talos dromaeo,Rev3 WINNT 6.1 shadow-central debug test reftest,Rev3 WINNT 6.1 maple talos tp4,Rev3 WINNT 6.1 cedar talos tp4,Rev3 WINNT 6.1 build-system opt test reftest,Rev3 WINNT 6.1 birch debug test mochitest-other,Rev3 WINNT 6.1 release-mozilla-central debug test mochitests-1/5,Rev3 WINNT 6.1 release-mozilla-central debug test mochitests-3/5,Rev3 WINNT 6.1 tracemonkey debug test mochitests-3/5,Rev3 WINNT 6.1 tracemonkey debug test mochitests-1/5,Rev3 WINNT 6.1 tracemonkey opt test reftest,Rev3 WINNT 6.1 birch debug test crashtest,Rev3 WINNT 6.1 mozilla-2.0 debug test crashtest,Rev3 WINNT 6.1 mozilla-central talos a11y,Rev3 WINNT 6.1 release-mozilla-central opt test jsreftest,Rev3 WINNT 6.1 tracemonkey opt test xpcshell,Rev3 WINNT 6.1 mozilla-central debug test reftest,Rev3 WINNT 6.1 mozilla-2.0 opt test mochitest-other,Rev3 WINNT 6.1 tryserver debug test mochitests-3/5,Rev3 WINNT 6.1 mozilla-central talos scroll,Rev3 WINNT 6.1 places debug test mochitests-5/5,Rev3 WINNT 6.1 places talos a11y,Rev3 WINNT 6.1 places debug test mochitests-3/5,Rev3 WINNT 6.1 maple debug test xpcshell,Rev3 WINNT 6.1 mozilla-2.0 talos dirty,Rev3 WINNT 6.1 jaegermonkey opt test mochitests-3/5,Rev3 WINNT 6.1 shadow-central opt test mochitest-other,Rev3 WINNT 6.1 jaegermonkey opt test mochitests-1/5,Rev3 WINNT 6.1 jaegermonkey debug test mochitest-other,Rev3 WINNT 6.1 cedar opt test jsreftest,Rev3 WINNT 6.1 release-mozilla-1.9.2 talos nochrome,Rev3 WINNT 6.1 mozilla-central debug test xpcshell,Rev3 WINNT 6.1 maple opt test reftest,Rev3 WINNT 6.1 tryserver debug test mochitest-other,Rev3 WINNT 6.1 maple debug test jsreftest,Rev3 WINNT 6.1 build-system opt test mochitests-4/5,Rev3 WINNT 6.1 shadow-central opt test crashtest,Rev3 WINNT 6.1 tryserver opt test reftest,Rev3 WINNT 6.1 mozilla-1.9.1 talos chrome,Rev3 WINNT 6.1 tracemonkey talos scroll,Rev3 WINNT 6.1 services-central talos dirty,Rev3 WINNT 6.1 jaegermonkey debug test jsreftest,Rev3 WINNT 6.1 mozilla-central opt test mochitests-4/5,Rev3 WINNT 6.1 jaegermonkey debug test mochitests-2/5,Rev3 WINNT 6.1 maple debug test mochitests-2/5,Rev3 WINNT 6.1 services-central debug test mochitests-4/5,Rev3 WINNT 6.1 places talos scroll,Rev3 WINNT 6.1 maple talos svg,Rev3 WINNT 6.1 release-mozilla-central opt test mochitests-4/5,Rev3 WINNT 6.1 maple talos dirty,Rev3 WINNT 6.1 maple opt test mochitests-3/5,Rev3 WINNT 6.1 release-mozilla-1.9.2 talos dromaeo,Rev3 WINNT 6.1 maple opt test mochitests-1/5,Rev3 WINNT 6.1 mozilla-1.9.2 talos scroll,Rev3 WINNT 6.1 mozilla-2.0 talos tp4,Rev3 WINNT 6.1 tracemonkey talos nochrome,Rev3 WINNT 6.1 birch talos scroll,Rev3 WINNT 6.1 tryserver talos tp4,Rev3 WINNT 6.1 birch opt test jsreftest,Rev3 WINNT 6.1 mozilla-central talos nochrome,Rev3 WINNT 6.1 services-central debug test crashtest,Rev3 WINNT 6.1 tryserver talos dirty,Rev3 WINNT 6.1 tracemonkey debug test xpcshell,Rev3 WINNT 6.1 mozilla-central talos svg,Rev3 WINNT 6.1 jaegermonkey opt test mochitest-other,Rev3 WINNT 6.1 cedar opt test mochitests-5/5,Rev3 WINNT 6.1 cedar opt test mochitests-3/5,Rev3 WINNT 6.1 places debug test reftest,Rev3 WINNT 6.1 birch debug test mochitests-2/5,Rev3 WINNT 6.1 cedar opt test xpcshell,Rev3 WINNT 6.1 release-mozilla-central talos dromaeo,Rev3 WINNT 6.1 birch opt test mochitests-4/5,Rev3 WINNT 6.1 mozilla-central opt test mochitests-5/5,Rev3 WINNT 6.1 mozilla-central debug test mochitests-5/5,Rev3 WINNT 6.1 cedar opt test reftest,Rev3 WINNT 6.1 tryserver talos nochrome,Rev3 WINNT 6.1 shadow-central opt test mochitests-4/5,Rev3 WINNT 6.1 mozilla-central debug test mochitests-4/5,Rev3 WINNT 6.1 build-system debug test mochitests-4/5,Rev3 WINNT 6.1 electrolysis opt test mochitests-2/5,Rev3 WINNT 6.1 build-system debug test mochitests-2/5,Rev3 WINNT 6.1 mozilla-2.0 debug test mochitests-5/5,Rev3 WINNT 6.1 birch debug test reftest,Rev3 WINNT 6.1 tryserver debug test reftest,Rev3 WINNT 6.1 tryserver opt test jsreftest,Rev3 WINNT 6.1 cedar debug test crashtest,Rev3 WINNT 6.1 services-central talos nochrome,Rev3 WINNT 6.1 maple debug test mochitest-other,Rev3 WINNT 6.1 mozilla-1.9.1 talos nochrome,Rev3 WINNT 6.1 shadow-central talos dromaeo,Rev3 WINNT 6.1 tracemonkey opt test mochitests-4/5,Rev3 WINNT 6.1 mozilla-2.0 opt test mochitests-4/5,Rev3 WINNT 6.1 mozilla-2.0 talos svg,Rev3 WINNT 6.1 mozilla-2.0 opt test mochitests-2/5,Rev3 WINNT 6.1 build-system talos scroll,Rev3 WINNT 6.1 jaegermonkey debug test crashtest,Rev3 WINNT 6.1 release-mozilla-central debug test xpcshell,Rev3 WINNT 6.1 build-system opt test mochitest-other,Rev3 WINNT 6.1 release-mozilla-central opt test reftest,Rev3 WINNT 6.1 release-mozilla-1.9.2 talos svg,Rev3 WINNT 6.1 tryserver opt test mochitests-4/5,Rev3 WINNT 6.1 shadow-central talos chrome,Rev3 WINNT 6.1 services-central debug test xpcshell,Rev3 WINNT 6.1 release-mozilla-central debug test mochitests-5/5,Rev3 WINNT 6.1 shadow-central debug test mochitests-1/5,Rev3 WINNT 6.1 services-central talos tp4,Rev3 WINNT 6.1 shadow-central talos nochrome,Rev3 WINNT 6.1 places opt test mochitest-other,Rev3 WINNT 6.1 build-system debug test mochitests-1/5,Rev3 WINNT 6.1 cedar opt test mochitest-other,Rev3 WINNT 6.1 tryserver talos dromaeo,Rev3 WINNT 6.1 cedar talos dirty,Rev3 WINNT 6.1 tryserver opt test xpcshell,Rev3 WINNT 6.1 services-central opt test mochitests-2/5,Rev3 WINNT 6.1 tracemonkey debug test mochitests-4/5,Rev3 WINNT 6.1 cedar talos svg,Rev3 WINNT 6.1 mozilla-2.0 talos scroll,Rev3 WINNT 6.1 maple opt test mochitest-other,Rev3 WINNT 6.1 mozilla-1.9.1 talos scroll,Rev3 WINNT 6.1 jaegermonkey opt test reftest,Rev3 WINNT 6.1 mozilla-central opt test mochitest-other,Rev3 WINNT 6.1 maple opt test xpcshell,Rev3 WINNT 6.1 electrolysis debug test mochitest-other,Rev3 WINNT 6.1 mozilla-2.0 talos a11y,Rev3 WINNT 6.1 shadow-central talos scroll,Rev3 WINNT 6.1 places talos svg,Rev3 WINNT 6.1 places debug test mochitests-4/5,Rev3 WINNT 6.1 electrolysis debug test crashtest,Rev3 WINNT 6.1 release-mozilla-1.9.2 talos scroll,Rev3 WINNT 6.1 tryserver debug test mochitests-5/5,Rev3 WINNT 6.1 jaegermonkey opt test mochitests-4/5,Rev3 WINNT 6.1 tryserver debug test mochitests-1/5,Rev3 WINNT 6.1 mozilla-2.0 opt test jsreftest,Rev3 WINNT 6.1 shadow-central talos a11y,Rev3 WINNT 6.1 places debug test crashtest,Rev3 WINNT 6.1 places opt test mochitests-4/5,Rev3 WINNT 6.1 cedar debug test mochitests-2/5,Rev3 WINNT 6.1 cedar debug test mochitests-4/5,Rev3 WINNT 6.1 places opt test mochitests-2/5,Rev3 WINNT 6.1 services-central opt test xpcshell,Rev3 WINNT 6.1 jaegermonkey debug test mochitests-5/5,Rev3 WINNT 6.1 cedar talos dromaeo,Rev3 WINNT 6.1 tracemonkey debug test reftest,Rev3 WINNT 6.1 maple opt test crashtest,Rev3 WINNT 6.1 build-system opt test mochitests-2/5,Rev3 WINNT 6.1 shadow-central talos svg,Rev3 WINNT 6.1 tracemonkey talos v8,Rev3 WINNT 6.1 cedar talos chrome,Rev3 WINNT 6.1 cedar talos nochrome,Rev3 WINNT 6.1 electrolysis debug test reftest,Rev3 WINNT 6.1 tryserver talos scroll,Rev3 WINNT 6.1 places debug test mochitests-1/5,Rev3 WINNT 6.1 services-central debug test mochitests-3/5,Rev3 WINNT 6.1 mozilla-central opt test xpcshell,Rev3 WINNT 6.1 jaegermonkey opt test jsreftest,Rev3 WINNT 6.1 release-mozilla-central opt test xpcshell,Rev3 WINNT 6.1 tracemonkey debug test mochitest-other,Rev3 WINNT 6.1 build-system talos dromaeo,Rev3 WINNT 6.1 tryserver opt test crashtest,Rev3 WINNT 6.1 release-mozilla-central opt test mochitest-other,Rev3 WINNT 6.1 mozilla-1.9.2 talos dirty,Rev3 WINNT 6.1 mozilla-2.0 talos dromaeo,Rev3 WINNT 6.1 cedar talos scroll,Rev3 WINNT 6.1 electrolysis opt test mochitest-other,Rev3 WINNT 6.1 mozilla-central opt test jsreftest,Rev3 WINNT 6.1 jaegermonkey opt test crashtest,Rev3 WINNT 6.1 release-mozilla-central opt test mochitests-2/5,Rev3 WINNT 6.1 maple opt test mochitests-5/5,Rev3 WINNT 6.1 electrolysis debug test mochitests-2/5,Rev3 WINNT 6.1 release-mozilla-central opt test crashtest,Rev3 WINNT 6.1 electrolysis debug test mochitests-4/5,Rev3 WINNT 6.1 release-mozilla-central talos scroll,Rev3 WINNT 6.1 mozilla-2.0 debug test jsreftest,Rev3 WINNT 6.1 maple talos a11y,Rev3 WINNT 6.1 places talos tp4,Rev3 WINNT 6.1 mozilla-central debug test crashtest,Rev3 WINNT 6.1 build-system debug test mochitest-other,Rev3 WINNT 6.1 birch opt test mochitests-2/5,Rev3 WINNT 6.1 cedar debug test reftest,Rev3 WINNT 6.1 cedar opt test mochitests-1/5,Rev3 WINNT 6.1 shadow-central opt test reftest,Rev3 WINNT 6.1 mozilla-central debug test mochitests-1/5,Rev3 WINNT 6.1 tryserver debug test crashtest,Rev3 WINNT 6.1 mozilla-2.0 debug test mochitests-2/5,Rev3 WINNT 6.1 cedar debug test xpcshell,Rev3 WINNT 6.1 mozilla-2.0 debug test mochitests-4/5,Rev3 WINNT 6.1 build-system debug test reftest,Rev3 WINNT 6.1 birch debug test mochitests-5/5,Rev3 WINNT 6.1 birch debug test mochitests-1/5,Rev3 WINNT 6.1 places debug test mochitest-other,Rev3 WINNT 6.1 build-system debug test xpcshell,Rev3 WINNT 6.1 shadow-central opt test mochitests-2/5,Rev3 WINNT 6.1 mozilla-2.0 opt test mochitests-3/5,Rev3 WINNT 6.1 tracemonkey opt test mochitests-3/5,Rev3 WINNT 6.1 mozilla-2.0 opt test mochitests-1/5,Rev3 WINNT 6.1 shadow-central talos dirty,Rev3 WINNT 6.1 mozilla-2.0 talos chrome,Rev3 WINNT 6.1 tryserver talos a11y,Rev3 WINNT 6.1 maple talos chrome,Rev3 WINNT 6.1 tryserver opt test mochitests-3/5,Rev3 WINNT 6.1 tryserver opt test mochitests-1/5,Rev3 WINNT 6.1 cedar debug test jsreftest,Rev3 WINNT 6.1 cedar debug test mochitest-other,Rev3 WINNT 6.1 mozilla-central debug test mochitest-other,Rev3 WINNT 6.1 shadow-central debug test mochitests-2/5,Rev3 WINNT 6.1 electrolysis opt test crashtest,Rev3 WINNT 6.1 shadow-central debug test mochitests-4/5,Rev3 WINNT 6.1 build-system talos chrome,Rev3 WINNT 6.1 mozilla-1.9.1 talos tp4,Rev3 WINNT 6.1 services-central opt test mochitests-4/5,Rev3 WINNT 6.1 release-mozilla-central debug test crashtest,Rev3 WINNT 6.1 shadow-central talos tp4,Rev3 WINNT 6.1 tracemonkey debug test jsreftest,Rev3 WINNT 6.1 services-central talos a11y,Rev3 WINNT 6.1 services-central opt test reftest,Rev3 WINNT 6.1 birch talos tp4,Rev3 WINNT 6.1 release-mozilla-1.9.2 talos tp4,Rev3 WINNT 6.1 birch opt test xpcshell,Rev3 WINNT 6.1 jaegermonkey debug test reftest,Rev3 WINNT 6.1 shadow-central debug test crashtest,Rev3 WINNT 6.1 mozilla-2.0 opt test reftest,Rev3 WINNT 6.1 release-mozilla-central debug test mochitests-2/5,Rev3 WINNT 6.1 tracemonkey debug test mochitests-2/5,Rev3 WINNT 6.1 mozilla-1.9.2 talos tp4,Rev3 WINNT 6.1 addonbaselinetester talos addon-baseline,Rev3 WINNT 6.1 tracemonkey opt test jsreftest,Rev3 WINNT 6.1 tryserver debug test mochitests-2/5,Rev3 WINNT 6.1 mozilla-central debug test jsreftest,Rev3 WINNT 6.1 birch opt test crashtest,Rev3 WINNT 6.1 jaegermonkey opt test mochitests-2/5,Rev3 WINNT 6.1 shadow-central debug test xpcshell,Rev3 WINNT 6.1 tracemonkey talos chrome,Rev3 WINNT 6.1 release-mozilla-central debug test mochitest-other,Rev3 WINNT 6.1 jaegermonkey opt test xpcshell,Rev3 WINNT 6.1 tryserver talos chrome>): [] 2011-03-08 08:14:30-0800 [Broker,41924,10.12.50.193] loop triggered while service disabled; ignoring trigger 2011-03-08 08:14:30-0800 [Broker,41924,10.12.50.193] BuildSlave.detached(talos-r3-w7-032) 2011-03-08 08:14:30-0800 [Broker,41921,10.12.50.88] BuildSlave.detached(talos-r3-snow-035) 2011-03-08 08:14:30-0800 [Broker,41864,10.12.49.218] BuildSlave.detached(talos-r3-fed64-038) 2011-03-08 08:14:30-0800 [Broker,41931,10.12.49.223] BuildSlave.detached(talos-r3-fed64-043) 2011-03-08 08:14:30-0800 [Broker,38147,10.12.50.156] BuildSlave.detached(talos-r3-xp-048) 2011-03-08 08:14:30-0800 [Broker,38147,10.12.50.156] <Build Rev3 WINNT 5.1 jaegermonkey opt test xpcshell>.lostRemote 2011-03-08 08:14:30-0800 [Broker,38147,10.12.50.156] stopping currentStep <buildbotcustom.steps.unittest.MozillaPackagedXPCShellTests instance at 0x654eb72c> 2011-03-08 08:14:30-0800 [Broker,38147,10.12.50.156] addCompleteLog(interrupt) 2011-03-08 08:14:30-0800 [Broker,38147,10.12.50.156] RemoteCommand.interrupt <RemoteShellCommand '['bash', '-c', 'if [ ! -d firefox/plugins ]; then mkdir firefox/plugins; fi && cp bin/xpcshell.exe firefox && cp -R bin/components/* firefox/components/ && cp -R bin/plugins/* firefox/plugins/ && python -u xpcshell/runxpcshelltests.py --symbols-path=symbols --manifest=xpcshell/tests/all-test-dirs.list firefox/xpcshell.exe']'> [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion. ] 2011-03-08 08:14:30-0800 [Broker,38147,10.12.50.156] RemoteCommand.disconnect: lost slave 2011-03-08 08:14:30-0800 [Broker,38147,10.12.50.156] Pulse <buildbotcustom.status.pulse.PulseStatus instance at 0x191ac50c>: Initiating push of 1 events 2011-03-08 08:14:30-0800 [Broker,38147,10.12.50.156] releaseLocks(<buildbotcustom.steps.unittest.MozillaPackagedXPCShellTests instance at 0x654eb72c>): [] 2011-03-08 08:14:30-0800 [Broker,38147,10.12.50.156] step 'xpcshell' complete: retry 2011-03-08 08:14:30-0800 [Broker,38147,10.12.50.156] <Build Rev3 WINNT 5.1 jaegermonkey opt test xpcshell>: build finished 2011-03-08 08:14:30-0800 [Broker,38147,10.12.50.156] sending mail (70475 bytes) to ['tinderbox-daemon@tinderbox.mozilla.org'] 2011-03-08 08:14:30-0800 [Broker,38147,10.12.50.156] Starting factory <twisted.mail.smtp.ESMTPSenderFactory instance at 0x6837c22c> 2011-03-08 08:14:35-0800 [Broker,38147,10.12.50.156] loop triggered while service disabled; ignoring trigger 2011-03-08 08:14:35-0800 [Broker,38147,10.12.50.156] loop triggered while service disabled; ignoring trigger 2011-03-08 08:14:35-0800 [Broker,41879,10.12.49.146] BuildSlave.detached(talos-r3-fed-019) 2011-03-08 08:14:35-0800 [Broker,41918,10.12.50.95] BuildSlave.detached(talos-r3-snow-042) 2011-03-08 08:14:35-0800 [Broker,41918,10.12.50.95] <Build Rev3 MacOSX Snow Leopard 10.6.2 build-system opt test mochitests-2/5>.lostRemote 2011-03-08 08:14:35-0800 [Broker,41918,10.12.50.95] stopping currentStep <buildbotcustom.steps.unittest.MozillaPackagedMochitests object at 0x2c91394c> 2011-03-08 08:14:35-0800 [Broker,41918,10.12.50.95] addCompleteLog(interrupt) 2011-03-08 08:14:35-0800 [Broker,41918,10.12.50.95] RemoteCommand.interrupt <RemoteShellCommand '['python', 'mochitest/runtests.py', '--appname=./Minefield.app/Contents/MacOS/firefox-bin', '--utility-path=bin', '--extra-profile-file=bin/plugins', '--certificate-path=certs', '--autorun', '--close-when-done', '--console-level=INFO', '--symbols-path=symbols', '--total-chunks', '5', '--this-chunk', '2', '--chunk-by-dir', '4']'> [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion. ] 2011-03-08 08:14:35-0800 [Broker,41918,10.12.50.95] RemoteCommand.disconnect: lost slave 2011-03-08 08:14:35-0800 [Broker,41918,10.12.50.95] releaseLocks(<buildbotcustom.steps.unittest.MozillaPackagedMochitests object at 0x2c91394c>): [] 2011-03-08 08:14:35-0800 [Broker,41918,10.12.50.95] step 'mochitest-plain-2' complete: retry 2011-03-08 08:14:35-0800 [Broker,41918,10.12.50.95] <Build Rev3 MacOSX Snow Leopard 10.6.2 build-system opt test mochitests-2/5>: build finished 2011-03-08 08:14:36-0800 [Broker,41918,10.12.50.95] sending mail (166894 bytes) to ['tinderbox-daemon@tinderbox.mozilla.org'] 2011-03-08 08:14:36-0800 [Broker,41918,10.12.50.95] Starting factory <twisted.mail.smtp.ESMTPSenderFactory instance at 0x3255d8cc> 2011-03-08 08:14:40-0800 [Broker,41918,10.12.50.95] loop triggered while service disabled; ignoring trigger 2011-03-08 08:14:40-0800 [Broker,41918,10.12.50.95] loop triggered while service disabled; ignoring trigger 2011-03-08 08:14:40-0800 [Broker,41860,10.12.49.202] BuildSlave.detached(talos-r3-fed64-022) 2011-03-08 08:14:40-0800 [Broker,41860,10.12.49.202] <Build Rev3 Fedora 12x64 tryserver debug test mochitests-4/5>.lostRemote 2011-03-08 08:14:40-0800 [Broker,41860,10.12.49.202] stopping currentStep <buildbotcustom.steps.unittest.MozillaPackagedMochitests object at 0x56835a4c> 2011-03-08 08:14:40-0800 [Broker,41860,10.12.49.202] addCompleteLog(interrupt) 2011-03-08 08:14:40-0800 [Broker,41860,10.12.49.202] RemoteCommand.interrupt <RemoteShellCommand '['python', 'mochitest/runtests.py', '--appname=firefox/firefox-bin', '--utility-path=bin', '--extra-profile-file=bin/plugins', '--certificate-path=certs', '--autorun', '--close-when-done', '--console-level=INFO', '--symbols-path=symbols', '--total-chunks', '5', '--this-chunk', '4', '--chunk-by-dir', '4']'> [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion. ] 2011-03-08 08:14:40-0800 [Broker,41860,10.12.49.202] RemoteCommand.disconnect: lost slave 2011-03-08 08:14:40-0800 [Broker,41860,10.12.49.202] releaseLocks(<buildbotcustom.steps.unittest.MozillaPackagedMochitests object at 0x56835a4c>): [] 2011-03-08 08:14:40-0800 [Broker,41860,10.12.49.202] step 'mochitest-plain-4' complete: retry 2011-03-08 08:14:40-0800 [Broker,41860,10.12.49.202] <Build Rev3 Fedora 12x64 tryserver debug test mochitests-4/5>: build finished 2011-03-08 08:14:41-0800 [Broker,41860,10.12.49.202] sending mail (144694 bytes) to ['tinderbox-daemon@tinderbox.mozilla.org'] 2011-03-08 08:14:41-0800 [Broker,41860,10.12.49.202] Starting factory <twisted.mail.smtp.ESMTPSenderFactory instance at 0x5cb6282c> 2011-03-08 08:14:45-0800 [Broker,41860,10.12.49.202] loop triggered while service disabled; ignoring trigger 2011-03-08 08:14:45-0800 [Broker,41860,10.12.49.202] loop triggered while service disabled; ignoring trigger 2011-03-08 08:14:45-0800 [Broker,41900,10.12.50.99] BuildSlave.detached(talos-r3-snow-046) 2011-03-08 08:14:46-0800 [Broker,41909,10.12.50.105] BuildSlave.detached(talos-r3-snow-052) 2011-03-08 08:14:46-0800 [Broker,41909,10.12.50.105] <Build Rev3 MacOSX Snow Leopard 10.6.2 build-system opt test reftest>.lostRemote 2011-03-08 08:14:46-0800 [Broker,41909,10.12.50.105] stopping currentStep <buildbotcustom.steps.unittest.MozillaPackagedReftests object at 0x6bca898c> 2011-03-08 08:14:46-0800 [Broker,41909,10.12.50.105] addCompleteLog(interrupt) 2011-03-08 08:14:46-0800 [Broker,41909,10.12.50.105] RemoteCommand.interrupt <RemoteShellCommand '['python', 'reftest/runreftest.py', '--appname=./Minefield.app/Contents/MacOS/firefox-bin', '--utility-path=bin', '--extra-profile-file=bin/plugins', '--symbols-path=symbols', 'reftest/tests/layout/reftests/reftest.list']'> [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion. ] 2011-03-08 08:14:46-0800 [Broker,41909,10.12.50.105] RemoteCommand.disconnect: lost slave 2011-03-08 08:14:46-0800 [Broker,41909,10.12.50.105] releaseLocks(<buildbotcustom.steps.unittest.MozillaPackagedReftests object at 0x6bca898c>): [] 2011-03-08 08:14:46-0800 [Broker,41909,10.12.50.105] step 'reftest' complete: retry 2011-03-08 08:14:46-0800 [Broker,41909,10.12.50.105] <Build Rev3 MacOSX Snow Leopard 10.6.2 build-system opt test reftest>: build finished 2011-03-08 08:14:46-0800 [Broker,41909,10.12.50.105] sending mail (154779 bytes) to ['tinderbox-daemon@tinderbox.mozilla.org'] 2011-03-08 08:14:46-0800 [Broker,41909,10.12.50.105] Starting factory <twisted.mail.smtp.ESMTPSenderFactory instance at 0x5d536f6c> 2011-03-08 08:14:51-0800 [Broker,41909,10.12.50.105] Received SIGTERM, shutting down. 2011-03-08 08:14:51-0800 [Broker,41909,10.12.50.105] loop triggered while service disabled; ignoring trigger 2011-03-08 08:14:51-0800 [Broker,41909,10.12.50.105] loop triggered while service disabled; ignoring trigger 2011-03-08 08:14:51-0800 [Broker,41911,10.12.50.104] BuildSlave.detached(talos-r3-snow-051) 2011-03-08 08:14:51-0800 [Broker,41911,10.12.50.104] <Build Rev3 MacOSX Snow Leopard 10.6.2 build-system talos dromaeo>.lostRemote 2011-03-08 08:14:51-0800 [Broker,41911,10.12.50.104] stopping currentStep <buildbotcustom.steps.talos.MozillaRunPerfTests instance at 0x6c1bc86c> 2011-03-08 08:14:51-0800 [Broker,41911,10.12.50.104] addCompleteLog(interrupt) 2011-03-08 08:14:51-0800 [Broker,41911,10.12.50.104] RemoteCommand.interrupt <RemoteShellCommand '['python', 'run_tests.py', '--noisy', '20110308_0804_config.yml']'> [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion. ] 2011-03-08 08:14:51-0800 [Broker,41911,10.12.50.104] RemoteCommand.disconnect: lost slave 2011-03-08 08:14:51-0800 [Broker,41911,10.12.50.104] releaseLocks(<buildbotcustom.steps.talos.MozillaRunPerfTests instance at 0x6c1bc86c>): [] 2011-03-08 08:14:51-0800 [Broker,41911,10.12.50.104] step 'Run performance tests' complete: retry 2011-03-08 08:14:51-0800 [Broker,41911,10.12.50.104] <Build Rev3 MacOSX Snow Leopard 10.6.2 build-system talos dromaeo>: build finished 2011-03-08 08:14:51-0800 [Broker,41911,10.12.50.104] sending mail (43698 bytes) to ['tinderbox-daemon@tinderbox.mozilla.org'] 2011-03-08 08:14:51-0800 [Broker,41911,10.12.50.104] Starting factory <twisted.mail.smtp.ESMTPSenderFactory instance at 0xc4ad0cc> 2011-03-08 08:14:56-0800 [Broker,41911,10.12.50.104] loop triggered while service disabled; ignoring trigger 2011-03-08 08:14:56-0800 [Broker,41911,10.12.50.104] loop triggered while service disabled; ignoring trigger 2011-03-08 08:14:56-0800 [Broker,41908,10.12.50.102] BuildSlave.detached(talos-r3-snow-049) 2011-03-08 08:14:56-0800 [Broker,41912,10.12.50.108] BuildSlave.detached(talos-r3-snow-055) 2011-03-08 08:14:56-0800 [Broker,41912,10.12.50.108] <Build Rev3 MacOSX Snow Leopard 10.6.2 build-system opt test mochitest-other>.lostRemote 2011-03-08 08:14:56-0800 [Broker,41912,10.12.50.108] stopping currentStep <buildbotcustom.steps.unittest.MozillaPackagedMochitests object at 0x54f82fcc> 2011-03-08 08:14:56-0800 [Broker,41912,10.12.50.108] addCompleteLog(interrupt) 2011-03-08 08:14:56-0800 [Broker,41912,10.12.50.108] RemoteCommand.interrupt <RemoteShellCommand '['python', 'mochitest/runtests.py', '--appname=./Minefield.app/Contents/MacOS/firefox-bin', '--utility-path=bin', '--extra-profile-file=bin/plugins', '--certificate-path=certs', '--autorun', '--close-when-done', '--console-level=INFO', '--symbols-path=symbols', '--browser-chrome']'> [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion. ] 2011-03-08 08:14:56-0800 [Broker,41912,10.12.50.108] RemoteCommand.disconnect: lost slave 2011-03-08 08:14:56-0800 [Broker,41912,10.12.50.108] releaseLocks(<buildbotcustom.steps.unittest.MozillaPackagedMochitests object at 0x54f82fcc>): [] 2011-03-08 08:14:56-0800 [Broker,41912,10.12.50.108] step 'mochitest-browser-chrome' complete: retry 2011-03-08 08:14:56-0800 [Broker,41912,10.12.50.108] <Build Rev3 MacOSX Snow Leopard 10.6.2 build-system opt test mochitest-other>: build finished 2011-03-08 08:14:57-0800 [Broker,41912,10.12.50.108] sending mail (363615 bytes) to ['tinderbox-daemon@tinderbox.mozilla.org'] 2011-03-08 08:14:57-0800 [Broker,41912,10.12.50.108] Starting factory <twisted.mail.smtp.ESMTPSenderFactory instance at 0x62fc22ac> 2011-03-08 08:15:02-0800 [Broker,41912,10.12.50.108] loop triggered while service disabled; ignoring trigger 2011-03-08 08:15:02-0800 [Broker,41912,10.12.50.108] loop triggered while service disabled; ignoring trigger 2011-03-08 08:15:02-0800 [Broker,41866,10.12.49.217] BuildSlave.detached(talos-r3-fed64-037) 2011-03-08 08:15:02-0800 [Broker,41866,10.12.49.217] <Build Rev3 Fedora 12x64 tryserver debug test mochitests-2/5>.lostRemote 2011-03-08 08:15:02-0800 [Broker,41866,10.12.49.217] stopping currentStep <buildbotcustom.steps.unittest.MozillaPackagedMochitests object at 0x2f6025cc> 2011-03-08 08:15:02-0800 [Broker,41866,10.12.49.217] addCompleteLog(interrupt) 2011-03-08 08:15:02-0800 [Broker,41866,10.12.49.217] RemoteCommand.interrupt <RemoteShellCommand '['python', 'mochitest/runtests.py', '--appname=firefox/firefox-bin', '--utility-path=bin', '--extra-profile-file=bin/plugins', '--certificate-path=certs', '--autorun', '--close-when-done', '--console-level=INFO', '--symbols-path=symbols', '--total-chunks', '5', '--this-chunk', '2', '--chunk-by-dir', '4']'> [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion. ] 2011-03-08 08:15:02-0800 [Broker,41866,10.12.49.217] RemoteCommand.disconnect: lost slave 2011-03-08 08:15:02-0800 [Broker,41866,10.12.49.217] releaseLocks(<buildbotcustom.steps.unittest.MozillaPackagedMochitests object at 0x2f6025cc>): [] 2011-03-08 08:15:02-0800 [Broker,41866,10.12.49.217] step 'mochitest-plain-2' complete: retry 2011-03-08 08:15:02-0800 [Broker,41866,10.12.49.217] <Build Rev3 Fedora 12x64 tryserver debug test mochitests-2/5>: build finished 2011-03-08 08:15:03-0800 [Broker,41866,10.12.49.217] sending mail (420540 bytes) to ['tinderbox-daemon@tinderbox.mozilla.org'] 2011-03-08 08:15:03-0800 [Broker,41866,10.12.49.217] Starting factory <twisted.mail.smtp.ESMTPSenderFactory instance at 0x2bdb89cc> 2011-03-08 08:15:08-0800 [Broker,41866,10.12.49.217] loop triggered while service disabled; ignoring trigger 2011-03-08 08:15:08-0800 [Broker,41866,10.12.49.217] loop triggered while service disabled; ignoring trigger 2011-03-08 08:15:08-0800 [Broker,41880,10.12.49.136] BuildSlave.detached(talos-r3-fed-009) 2011-03-08 08:15:08-0800 [Broker,41929,10.12.50.96] BuildSlave.detached(talos-r3-snow-043) 2011-03-08 08:15:08-0800 [Broker,41907,10.12.50.90] BuildSlave.detached(talos-r3-snow-037) 2011-03-08 08:15:08-0800 [Broker,41917,10.12.50.98] BuildSlave.detached(talos-r3-snow-045) 2011-03-08 08:15:08-0800 [Broker,41917,10.12.50.98] <Build Rev3 MacOSX Snow Leopard 10.6.2 build-system talos chrome>.lostRemote 2011-03-08 08:15:08-0800 [Broker,41917,10.12.50.98] stopping currentStep <buildbotcustom.steps.talos.MozillaRunPerfTests instance at 0x6287080c> 2011-03-08 08:15:08-0800 [Broker,41917,10.12.50.98] addCompleteLog(interrupt) 2011-03-08 08:15:08-0800 [Broker,41917,10.12.50.98] RemoteCommand.interrupt <RemoteShellCommand '['python', 'run_tests.py', '--noisy', '20110308_0804_config.yml']'> [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion. ] 2011-03-08 08:15:08-0800 [Broker,41917,10.12.50.98] RemoteCommand.disconnect: lost slave 2011-03-08 08:15:08-0800 [Broker,41917,10.12.50.98] releaseLocks(<buildbotcustom.steps.talos.MozillaRunPerfTests instance at 0x6287080c>): [] 2011-03-08 08:15:08-0800 [Broker,41917,10.12.50.98] step 'Run performance tests' complete: retry 2011-03-08 08:15:08-0800 [Broker,41917,10.12.50.98] <Build Rev3 MacOSX Snow Leopard 10.6.2 build-system talos chrome>: build finished 2011-03-08 08:15:09-0800 [Broker,41917,10.12.50.98] sending mail (214494 bytes) to ['tinderbox-daemon@tinderbox.mozilla.org'] 2011-03-08 08:15:09-0800 [Broker,41917,10.12.50.98] Starting factory <twisted.mail.smtp.ESMTPSenderFactory instance at 0x1a4dfd0c> 2011-03-08 08:15:14-0800 [Broker,41917,10.12.50.98] loop triggered while service disabled; ignoring trigger 2011-03-08 08:15:14-0800 [Broker,41917,10.12.50.98] loop triggered while service disabled; ignoring trigger 2011-03-08 08:15:14-0800 [Broker,41904,10.12.49.174] BuildSlave.detached(talos-r3-fed-047) 2011-03-08 08:15:14-0800 [Broker,41915,10.12.50.97] BuildSlave.detached(talos-r3-snow-044) 2011-03-08 08:15:14-0800 [Broker,41915,10.12.50.97] <Build Rev3 MacOSX Snow Leopard 10.6.2 build-system talos nochrome>.lostRemote 2011-03-08 08:15:14-0800 [Broker,41915,10.12.50.97] stopping currentStep <buildbotcustom.steps.talos.MozillaRunPerfTests instance at 0x67c2970c> 2011-03-08 08:15:14-0800 [Broker,41915,10.12.50.97] addCompleteLog(interrupt) 2011-03-08 08:15:14-0800 [Broker,41915,10.12.50.97] RemoteCommand.interrupt <RemoteShellCommand '['python', 'run_tests.py', '--noisy', '20110308_0806_config.yml']'> [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion. ] 2011-03-08 08:15:14-0800 [Broker,41915,10.12.50.97] RemoteCommand.disconnect: lost slave 2011-03-08 08:15:14-0800 [Broker,41915,10.12.50.97] releaseLocks(<buildbotcustom.steps.talos.MozillaRunPerfTests instance at 0x67c2970c>): [] 2011-03-08 08:15:14-0800 [Broker,41915,10.12.50.97] step 'Run performance tests' complete: retry 2011-03-08 08:15:14-0800 [Broker,41915,10.12.50.97] <Build Rev3 MacOSX Snow Leopard 10.6.2 build-system talos nochrome>: build finished 2011-03-08 08:15:14-0800 [Broker,41915,10.12.50.97] sending mail (47575 bytes) to ['tinderbox-daemon@tinderbox.mozilla.org'] 2011-03-08 08:15:14-0800 [Broker,41915,10.12.50.97] Starting factory <twisted.mail.smtp.ESMTPSenderFactory instance at 0x2f68914c> 2011-03-08 08:15:19-0800 [Broker,41915,10.12.50.97] loop triggered while service disabled; ignoring trigger 2011-03-08 08:15:19-0800 [Broker,41915,10.12.50.97] loop triggered while service disabled; ignoring trigger 2011-03-08 08:15:19-0800 [Broker,40842,10.12.50.229] BuildSlave.detached(t-r3-w764-016) 2011-03-08 08:15:19-0800 [-] adbapi closing: MySQLdb 2011-03-08 08:15:19-0800 [-] adbapi closing: MySQLdb 2011-03-08 08:15:19-0800 [-] adbapi closing: MySQLdb
The master is back up and it is now giving jobs to slaves. I wonder why we had not noticed before.
Summary: buildbot-master2:8011 seems to be down → buildbot-master2:8011 had been down since yesterday (2011-03-08 08:15)
I need your expertise to figure this out. It seems the master went rogue after yesterday's reconfig. I will attach a joint log.
Priority: P1 → P2
45 minutes after yesterday's reconfig the master stopped. > 2011-03-08 07:30:07-0800 [-] configuration update started > 2011-03-08 07:32:46-0800 [-] configuration update complete > 2011-03-08 08:15:19-0800 [-] adbapi closing: MySQLdb I don't know if we have nagios coverage there but I noticed something weird. When I run "ps aux | grep bot" I could actually see test_master5 listed there so I would not be surprised if nagios could not notice it. When I run that command a second time I could not see tests_master5 listed there anymore. Before stopping the master I can see a lot of ConnectionLost. Just before the first ConnectionLost we can see this. 2011-03-08 08:11:14-0800 [-] Pulse <0x1aff8c2c>: shutting down 2011-03-08 08:11:14-0800 [-] Pulse <0x1aff8c2c>: Initiating push of 3 events 2011-03-08 08:11:14-0800 [-] Stopping thread pool <twisted.python.threadpool.ThreadPool instance at 0x6873c6cc> 2011-03-08 08:11:14-0800 [-] Stopping thread pool <twisted.python.threadpool.ThreadPool instance at 0x60412dec> 2011-03-08 08:11:14-0800 [-] Stopping thread pool <twisted.python.threadpool.ThreadPool instance at 0x61c02dec> 2011-03-08 08:11:14-0800 [-] Stopping thread pool <twisted.python.threadpool.ThreadPool instance at 0x55f386ac> 2011-03-08 08:11:14-0800 [-] Stopping thread pool <twisted.python.threadpool.ThreadPool instance at 0x6804548c> 2011-03-08 08:11:14-0800 [-] Stopping thread pool <twisted.python.threadpool.ThreadPool instance at 0x6906bd8c> 2011-03-08 08:11:14-0800 [-] Stopping thread pool <twisted.python.threadpool.ThreadPool instance at 0x68145a0c> 2011-03-08 08:11:14-0800 [-] Stopping thread pool <twisted.python.threadpool.ThreadPool instance at 0x5f673fcc> 2011-03-08 08:11:14-0800 [-] Stopping thread pool <twisted.python.threadpool.ThreadPool instance at 0x6845e3ec> 2011-03-08 08:11:14-0800 [-] Stopping thread pool <twisted.python.threadpool.ThreadPool instance at 0x69e834cc> 2011-03-08 08:11:14-0800 [-] Stopping thread pool <twisted.python.threadpool.ThreadPool instance at 0x67aff66c> 2011-03-08 08:11:14-0800 [-] Stopping thread pool <twisted.python.threadpool.ThreadPool instance at 0x5f3b498c> 2011-03-08 08:11:14-0800 [-] Stopping thread pool <twisted.python.threadpool.ThreadPool instance at 0x5fdc98cc> 2011-03-08 08:11:14-0800 [-] Stopping thread pool <twisted.python.threadpool.ThreadPool instance at 0x6033188c> 2011-03-08 08:11:14-0800 [-] Stopping thread pool <twisted.python.threadpool.ThreadPool instance at 0x1778392c> 2011-03-08 08:11:14-0800 [-] Stopping thread pool <twisted.python.threadpool.ThreadPool instance at 0x3d19256c> 2011-03-08 08:11:14-0800 [-] Stopping thread pool <twisted.python.threadpool.ThreadPool instance at 0xbe8d38c> 2011-03-08 08:11:14-0800 [-] Stopping thread pool <twisted.python.threadpool.ThreadPool instance at 0xbe6922c> 2011-03-08 08:11:14-0800 [-] Stopping thread pool <twisted.python.threadpool.ThreadPool instance at 0xbdec54c> 2011-03-08 08:11:14-0800 [-] Stopping thread pool <twisted.python.threadpool.ThreadPool instance at 0xb21548c> 2011-03-08 08:11:15-0800 [-] Pulse <0x1aff8c2c>: Processed 3, sent 3 events (0 heartbeats) in 0.34 seconds 2011-03-08 08:13:33-0800 [Broker,41915,10.12.50.97] <RemoteShellCommand '['python', 'run_tests.py', '--noisy', '20110308_0806_config.yml']'> rc=0 2011-03-08 08:13:33-0800 [Broker,41916,10.12.50.91] <RemoteShellCommand '['python', 'run_tests.py', '--noisy', '20110308_0806_config.yml']'> rc=0 2011-03-08 08:13:33-0800 [-] (Port 8011 Closed) 2011-03-08 08:13:33-0800 [-] Stopping factory <buildbot.status.web.baseweb.RotateLogSite instance at 0x5fdcf42c> 2011-03-08 08:13:33-0800 [-] (Port 9011 Closed) 2011-03-08 08:13:33-0800 [-] Stopping factory <twisted.spread.pb.PBServerFactory instance at 0x9e6f4ec> 2011-03-08 08:13:33-0800 [-] (Port 1236 Closed) 2011-03-08 08:13:33-0800 [-] Stopping factory <twisted.conch.manhole_ssh.ConchFactory instance at 0xac1fc2c> I can also see SIGTERM on the log. Is it related? (I see two of them).
2011-03-08 08:11:00-0800 [-] Received SIGTERM, shutting down. Someone shut the master down yesterday at 8:11am. Maybe it was an errant 'make stop' in the wrong directly? At any rate, there's no technical mystery here..
(In reply to comment #1) > The master is back up and it is now giving jobs to slaves. Phew. > I wonder why we had not noticed before. Yes, this is a worry. How many slaves were offline, and for how long, because of this?
Whiteboard: [badmaster] → [badmaster][slaveduty]
(In reply to comment #5) > > I wonder why we had not noticed before. > Yes, this is a worry. How many slaves were offline, and for how long, because > of this? 45 slaves with my hand counting. http://buildbot-master2.build.mozilla.org:8011/buildslaves?no_builders=1
It seems that this was just few chained unlucky events. The master was shutdown by mistake and the nagios check was ack'ed incorrectly. As we can see in the comment bear thought it was acknowledging the right master (bm1) which was shut down that morning. FTR when we disable a master we modify the /etc/nagios/nrpe.cfg and we call |/etc/init.d/nrpe restart|. That way nagios won't complain that there is a different number of processes running than expected. [2011-03-08 08:35:21] <nagios> [36] buildbot-master2.build.scl1:buildbot is 4CRITICAL: PROCS CRITICAL: 3 processes with command name buildbot [2011-03-08 08:39:25] <nagios> buildbot-master2.build.scl1:buildbot is 12ACKNOWLEDGEMENT (CRITICAL): PROCS CRITICAL: 3 processes with command name buildbot;bear|buildduty;tests_master5 shutdown on purpose
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: