Exceptions in try masters

RESOLVED FIXED

Status

RESOLVED FIXED
8 years ago
5 years ago

People

(Reporter: catlee, Assigned: aki)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [try][mobile])

Attachments

(1 attachment, 1 obsolete attachment)

(Reporter)

Description

8 years ago
Since Wednesday sometime we've been getting these exceptions in the try masters:
Exception in /builds/buildbot/try_master1/master/twistd.log:
2011-04-07 19:28:38-0700 [-] Unhandled Error
	Traceback (most recent call last):
	  File "/builds/buildbot/try_master1/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/base.py", line 374, in _stepDone
	    return self.startNextStep()
	  File "/builds/buildbot/try_master1/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/base.py", line 363, in startNextStep
	    d = defer.maybeDeferred(s.startStep, self.remote)
	  File "/builds/buildbot/try_master1/lib/python2.6/site-packages/twisted/internet/defer.py", line 125, in maybeDeferred
	    result = f(*args, **kw)
	  File "/builds/buildbot/try_master1/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/buildstep.py", line 724, in startStep
	    self.step_status.setText(self.describe(False))
	--- <exception caught here> ---
	  File "/builds/buildbot/try_master1/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/steps/shell.py", line 144, in describe
	    words = properties.render(words)
	  File "/builds/buildbot/try_master1/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/properties.py", line 108, in render
	    return [ self.render(e) for e in value ]
	  File "/builds/buildbot/try_master1/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/properties.py", line 106, in render
	    return value.render(self.pmap)
	  File "/builds/buildbot/try_master1/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/properties.py", line 194, in render
	    s = self.fmtstring % pmap
	  File "/builds/buildbot/try_master1/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/properties.py", line 169, in __getitem__
	    rv = properties[key]
	  File "/builds/buildbot/try_master1/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/properties.py", line 50, in __getitem__
	    rv = self.properties[name][0]
	exceptions.KeyError: 'mobile_repo'
(Assignee)

Comment 1

8 years ago
To reconfig try on buildbot-master5 or production-master02, I pretty much am forced to gracefully shut down the master and restart.
(Assignee)

Comment 2

8 years ago
--------------------------------------------------------------------------------
Exception in /builds/buildbot/tests_master5/master/twistd.log:
2011-04-11 17:58:54-0700 [Broker,14517,10.12.49.234] Unhandled Error
	Traceback (most recent call last):
	  File "/tools/python-2.6.5/lib/python2.6/threading.py", line 484, in run
	    self.__target(*self.__args, **self.__kwargs)
	  File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/twisted/python/threadpool.py", line 207, in _worker
	    result = context.call(ctx, function, *args, **kwargs)
	  File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/twisted/python/context.py", line 59, in callWithContext
	    return self.currentContext().callWithContext(ctx, func, *args, **kw)
	  File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/twisted/python/context.py", line 37, in callWithContext
	    return func(*args,**kw)
	--- <exception caught here> ---
	  File "/builds/buildbot/tests_master5/lib/python2.6/site-packages/buildbotcustom/status/log_handlers.py", line 95, in handleLogs
	    subprocess.check_call(cmd, stdout=output, stderr=subprocess.STDOUT)
	  File "/tools/python-2.6.5/lib/python2.6/subprocess.py", line 493, in check_call
	    retcode = call(*popenargs, **kwargs)
	  File "/tools/python-2.6.5/lib/python2.6/subprocess.py", line 480, in call
	    return Popen(*popenargs, **kwargs).wait()
	  File "/tools/python-2.6.5/lib/python2.6/subprocess.py", line 633, in __init__
	    errread, errwrite)
	  File "/tools/python-2.6.5/lib/python2.6/subprocess.py", line 1049, in _execute_child
	    self.pid = os.fork()
	exceptions.OSError: [Errno 12] Cannot allocate memory
(Assignee)

Comment 3

8 years ago
I'm going to guess the error in comment 0 has something to do with doStepIf not respecting haltOnFailure in a previous step, but am not sure.
(Assignee)

Comment 4

8 years ago
Created attachment 525280 [details] [diff] [review]
fix try exceptions ?
Attachment #525280 - Flags: review?(catlee)
(Assignee)

Comment 5

8 years ago
Created attachment 525283 [details] [diff] [review]
fix descriptions
Attachment #525280 - Attachment is obsolete: true
Attachment #525280 - Flags: review?(catlee)
Attachment #525283 - Flags: review?(catlee)
(Reporter)

Updated

8 years ago
Attachment #525283 - Flags: review?(catlee) → review+
(Assignee)

Updated

8 years ago
Assignee: nobody → aki
(Assignee)

Comment 7

8 years ago
This appears fixed on pm02-try, which was restarted.
Still get them from bm3-try, which I'm going to attribute to needing another reconfig or restart.
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.