641788 - buildbot throws lots of Exceptions when a Build's logs disappear before it finishes

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect, P3)

Description

[[e-mail] (:bhearsum)]

We see this case whenever a slave gets hung mid-job and we don't notice until after a something has cleaned up its master-side logs. There's a few Exceptions I've seen like this. Here's a couple:
        Traceback (most recent call last):
          File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/twisted/internet/defer.py", line 249, in addCallbacks 
            self._runCallbacks()
          File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks
            self.result = callback(self.result, *args, **kw)
          File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/twisted/internet/defer.py", line 318, in callback
            self._startRunCallbacks(result)
          File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/twisted/internet/defer.py", line 424, in _startRunCallbacks
            self._runCallbacks()
        --- <exception caught here> --- 
          File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks
            self.result = callback(self.result, *args, **kw)
          File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/buildstep.py", line 1064, in <lambda>
            d.addCallback(lambda res: self.createSummary(cmd.logs['stdio']))
          File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/buildbotcustom/steps/unittest.py", line 808, in createSummary
            
          File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/buildbotcustom/steps/unittest.py", line 144, in summarizeLogJetpacktests
            for line in log.readlines():
          File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/status/builder.py", line 400, in readlines
            alltext = "".join(self.getChunks([channel], onlyText=True))
          File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/status/builder.py", line 339, in getChunks
            f = self.getFile()
          File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/status/builder.py", line 317, in getFile
            return open(self.getFilename(), "r")
        exceptions.IOError: [Errno 2] No such file or directory: './mozilla-central_leopard-o-debug_test-jetpack/343-log-jetpack-stdio'
        


2011-03-13 19:51:48-0700 [-] Unhandled Error
	Traceback (most recent call last):
	  File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks
	    self.result = callback(self.result, *args, **kw)
	  File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/twisted/internet/defer.py", line 318, in callback
	    self._startRunCallbacks(result)
	  File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/twisted/internet/defer.py", line 424, in _startRunCallbacks
	    self._runCallbacks()
	  File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks
	    self.result = callback(self.result, *args, **kw)
	--- <exception caught here> ---
	  File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/buildstep.py", line 885, in failed
	    self.step_status.stepFinished(EXCEPTION)
	  File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/status/builder.py", line 1047, in stepFinished
	    if os.path.getsize(loog.getFilename()) > logCompressionLimit:
	  File "/builds/buildbot/tests_master6/lib/python2.6/genericpath.py", line 49, in getsize
	    return os.stat(filename).st_size
	exceptions.OSError: [Errno 2] No such file or directory: './mozilla-central_leopard-o-debug_test-jetpack/343-log-jetpack-stdio'

Upstream ticket is http://trac.buildbot.net/ticket/1881

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

We could also make the master cleanup script smarter, maybe open the pickle and look for signs the job is finished.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

N/M, that's bug 641809.

Comment 3

[[e-mail] (:bhearsum)]

Not worth fixing since we're moving to Taskcluster.