648475 - Exceptions in try masters

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Chris AtLee [:catlee]]

Since Wednesday sometime we've been getting these exceptions in the try masters:
Exception in /builds/buildbot/try_master1/master/twistd.log:
2011-04-07 19:28:38-0700 [-] Unhandled Error
	Traceback (most recent call last):
	  File "/builds/buildbot/try_master1/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/base.py", line 374, in _stepDone
	    return self.startNextStep()
	  File "/builds/buildbot/try_master1/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/base.py", line 363, in startNextStep
	    d = defer.maybeDeferred(s.startStep, self.remote)
	  File "/builds/buildbot/try_master1/lib/python2.6/site-packages/twisted/internet/defer.py", line 125, in maybeDeferred
	    result = f(*args, **kw)
	  File "/builds/buildbot/try_master1/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/buildstep.py", line 724, in startStep
	    self.step_status.setText(self.describe(False))
	--- <exception caught here> ---
	  File "/builds/buildbot/try_master1/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/steps/shell.py", line 144, in describe
	    words = properties.render(words)
	  File "/builds/buildbot/try_master1/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/properties.py", line 108, in render
	    return [ self.render(e) for e in value ]
	  File "/builds/buildbot/try_master1/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/properties.py", line 106, in render
	    return value.render(self.pmap)
	  File "/builds/buildbot/try_master1/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/properties.py", line 194, in render
	    s = self.fmtstring % pmap
	  File "/builds/buildbot/try_master1/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/properties.py", line 169, in __getitem__
	    rv = properties[key]
	  File "/builds/buildbot/try_master1/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/properties.py", line 50, in __getitem__
	    rv = self.properties[name][0]
	exceptions.KeyError: 'mobile_repo'

Comment 1

[Aki Sasaki (not active)]

To reconfig try on buildbot-master5 or production-master02, I pretty much am forced to gracefully shut down the master and restart.

Comment 2

[Aki Sasaki (not active)]

--------------------------------------------------------------------------------
Exception in /builds/buildbot/tests_master5/master/twistd.log:
2011-04-11 17:58:54-0700 [Broker,14517,10.12.49.234] Unhandled Error
	Traceback (most recent call last):
	  File "/tools/python-2.6.5/lib/python2.6/threading.py", line 484, in run
	    self.__target(*self.__args, **self.__kwargs)
	  File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/twisted/python/threadpool.py", line 207, in _worker
	    result = context.call(ctx, function, *args, **kwargs)
	  File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/twisted/python/context.py", line 59, in callWithContext
	    return self.currentContext().callWithContext(ctx, func, *args, **kw)
	  File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/twisted/python/context.py", line 37, in callWithContext
	    return func(*args,**kw)
	--- <exception caught here> ---
	  File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/buildbotcustom/status/log_handlers.py", line 95, in handleLogs
	    subprocess.check_call(cmd, stdout=output, stderr=subprocess.STDOUT)
	  File "/tools/python-2.6.5/lib/python2.6/subprocess.py", line 493, in check_call
	    retcode = call(*popenargs, **kwargs)
	  File "/tools/python-2.6.5/lib/python2.6/subprocess.py", line 480, in call
	    return Popen(*popenargs, **kwargs).wait()
	  File "/tools/python-2.6.5/lib/python2.6/subprocess.py", line 633, in __init__
	    errread, errwrite)
	  File "/tools/python-2.6.5/lib/python2.6/subprocess.py", line 1049, in _execute_child
	    self.pid = os.fork()
	exceptions.OSError: [Errno 12] Cannot allocate memory

Comment 3

[Aki Sasaki (not active)]

I'm going to guess the error in comment 0 has something to do with doStepIf not respecting haltOnFailure in a previous step, but am not sure.

Comment 4

[Aki Sasaki (not active)]

Attachment: fix try exceptions ?

Comment 5

[Aki Sasaki (not active)]

Attachment: fix descriptions

Comment 6

[Aki Sasaki (not active)]

Comment on attachment 525283 [details] [diff] [review]
fix descriptions

http://hg.mozilla.org/build/buildbotcustom/rev/ea3c5f528366

Comment 7

[Aki Sasaki (not active)]

This appears fixed on pm02-try, which was restarted.
Still get them from bm3-try, which I'm going to attribute to needing another reconfig or restart.