786227 - rashes of KeyError's coming out of various masters since turning of android xul / 10.5 stuff yesterday

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[[e-mail] (:bhearsum)]

An example full Traceback:
2012-08-28 02:47:37-0700 [Broker,173347,10.12.50.185] Unhandled Error
	Traceback (most recent call last):
	  File "/builds/buildbot/tests1-windows/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/process/buildstep.py", line 875, in finished
	    self.step_status.stepFinished(results)
	  File "/builds/buildbot/tests1-windows/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/status/builder.py", line 1077, in stepFinished
	    w.callback(self)
	  File "/builds/buildbot/tests1-windows/lib/python2.6/site-packages/twisted/internet/defer.py", line 318, in callback
	    self._startRunCallbacks(result)
	  File "/builds/buildbot/tests1-windows/lib/python2.6/site-packages/twisted/internet/defer.py", line 424, in _startRunCallbacks
	    self._runCallbacks()
	--- <exception caught here> ---
	  File "/builds/buildbot/tests1-windows/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks
	    self.result = callback(self.result, *args, **kw)
	  File "/builds/buildbot/tests1-windows/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/status/builder.py", line 1426, in _stepFinished
	    w.stepFinished(self, step, results)
	  File "/builds/buildbot/tests1-windows/buildbotcustom/status/pulse.py", line 226, in stepFinished
	    builderName = escape(self._translateBuilderName(build.builder.name))
	  File "/builds/buildbot/tests1-windows/buildbotcustom/status/pulse.py", line 166, in _translateBuilderName
	    builder = self.status.getBuilder(builderName)
	  File "/builds/buildbot/tests1-windows/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/status/builder.py", line 2413, in getBuilder
	    return self.botmaster.builders[name].builder_status
	exceptions.KeyError: 'Rev3 WINNT 6.1 mozilla-beta pgo talos tprow'


And some other short ones:
	exceptions.KeyError: 'OS X 10.7 32-bit try leak test build'
	exceptions.KeyError: 'OS X 10.7 32-bit mozilla-aurora leak test build'
	exceptions.KeyError: 'Rev3 MacOSX Leopard 10.5.8 mozilla-beta talos tprow'
	exceptions.KeyError: 'Rev3 WINNT 5.1 mozilla-beta pgo talos tprow'
	exceptions.KeyError: 'Rev4 MacOSX Lion 10.7 mozilla-beta talos tprow'


Someone suggested that this could be people trying to trigger builders that no longer exist, so they might just go away. Pulse is also in the traceback, so maybe we need to do a one of clean up of some things?

Comment 1

[Chris AtLee [:catlee]]

One hypothesis: this is caused by currently running builds for builders that have been removed. for each step that finishes, the pulse status plugin gets run, which then tries to get the builder object from the master. The builder object is no longer accessible by name, so you get the key error.

It may be possible to get at the builder object another way (e.g. via a reference in the step/build object).

Comment 2

[[e-mail] (:bhearsum)]

Well, haven't seen any more of these today so I'm 95% sure it was a transient issue.