995146 - Reconfig bustage - temporary fix for "exceptions.KeyError: 'tst-linux64-ec2-300'"

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Pete Moore [:pmoore][:pete]]

During reconfig, got:

    2014-04-11 02:02:35-0700 [-] Creating BuildMaster -- buildbot.version: 0.8.2-hg-ca197cab42a9-production-0.8
    2014-04-11 02:02:35-0700 [-] loading configuration from /builds/buildbot/tests1-linux64/master/master.cfg
    2014-04-11 02:02:35-0700 [-] JacuzziAllocator 64528144: created
    2014-04-11 02:02:35-0700 [-] nextAWSSlave: start
    2014-04-11 02:02:35-0700 [-] nextAWSSlave: start
    2014-04-11 02:02:46-0700 [-] finished loading config file
    2014-04-11 02:03:00-0700 [-] configuration update started
    2014-04-11 02:03:00-0700 [-] configuration update failed
    2014-04-11 02:03:00-0700 [-] Unhandled Error
            Traceback (most recent call last):
              File "/builds/buildbot/tests1-linux64/lib/python2.7/site-packages/buildbot-0.8.2_hg_ca197cab42a9_production_0.8-py2.7.egg/buildbot/master.py", line 628, in loadTheConfigFile
                d = self.loadConfig(f)
              File "/builds/buildbot/tests1-linux64/lib/python2.7/site-packages/buildbot-0.8.2_hg_ca197cab42a9_production_0.8-py2.7.egg/buildbot/master.py", line 939, in loadConfig
                d.addCallback(lambda res: self.loadConfig_Slaves(slaves))
              File "/builds/buildbot/tests1-linux64/lib/python2.7/site-packages/twisted/internet/defer.py", line 260, in addCallback
                callbackKeywords=kw)
              File "/builds/buildbot/tests1-linux64/lib/python2.7/site-packages/twisted/internet/defer.py", line 249, in addCallbacks
                self._runCallbacks()
            --- <exception caught here> ---
              File "/builds/buildbot/tests1-linux64/lib/python2.7/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks
                self.result = callback(self.result, *args, **kw)
              File "/builds/buildbot/tests1-linux64/lib/python2.7/site-packages/buildbot-0.8.2_hg_ca197cab42a9_production_0.8-py2.7.egg/buildbot/master.py", line 939, in <lambda>
                d.addCallback(lambda res: self.loadConfig_Slaves(slaves))
              File "/builds/buildbot/tests1-linux64/lib/python2.7/site-packages/buildbot-0.8.2_hg_ca197cab42a9_production_0.8-py2.7.egg/buildbot/master.py", line 1065, in loadConfig_Slaves
                return self.botmaster.loadConfig_Slaves(new_slaves)
              File "/builds/buildbot/tests1-linux64/lib/python2.7/site-packages/buildbot-0.8.2_hg_ca197cab42a9_production_0.8-py2.7.egg/buildbot/master.py", line 236, in loadConfig_Slaves
                dl.append(self.removeSlave(s))
              File "/builds/buildbot/tests1-linux64/lib/python2.7/site-packages/buildbot-0.8.2_hg_ca197cab42a9_production_0.8-py2.7.egg/buildbot/master.py", line 254, in removeSlave
                d = self.slaves[s.slavename].disconnect()
            exceptions.KeyError: 'tst-linux64-ec2-300' 


Submit a correction or amendment below (click here to make a fresh posting)

http://hg.mozilla.org/build/buildbot-configs/rev/dbc0e8a93cb7#l3.33 vs http://hg.mozilla.org/build/buildbot-configs/rev/dbc0e8a93cb7#l3.112

Comment 1

[Pete Moore [:pmoore][:pete]]

Attachment: buildbot-configs patch

Temporary fix to get masters working again

Comment 2

[Pete Moore [:pmoore][:pete]]

https://hg.mozilla.org/build/buildbot-configs/rev/6cf12122077e

Comment 3

[Pete Moore [:pmoore][:pete]]

Running a new reconfig...

Comment 4

[Pete Moore [:pmoore][:pete]]

Reconfig successful.

Comment 5

[[e-mail] (:bhearsum)]

This is really weird. We've dropped slaves plently of times and never hit this. Perhaps something was connected under that name, even though AFAICT, it doesn't exist in AWS.

Comment 6

[Pete Moore [:pmoore][:pete]]

I had this for the following 9 masters:

reconfig_1397206657.log:[buildbot-master51.srv.releng.use1.mozilla.com] out: 	exceptions.KeyError: 'tst-linux64-ec2-300'
reconfig_1397206657.log:[buildbot-master52.srv.releng.use1.mozilla.com] out: 	exceptions.KeyError: 'tst-linux64-ec2-300'
reconfig_1397206657.log:[buildbot-master53.srv.releng.usw2.mozilla.com] out: 	exceptions.KeyError: 'tst-linux64-ec2-300'
reconfig_1397206657.log:[buildbot-master54.srv.releng.usw2.mozilla.com] out: 	exceptions.KeyError: 'tst-linux64-ec2-300'
reconfig_1397206657.log:[buildbot-master67.srv.releng.use1.mozilla.com] out: 	exceptions.KeyError: 'tst-linux64-ec2-300'
reconfig_1397206657.log:[buildbot-master68.srv.releng.usw2.mozilla.com] out: 	exceptions.KeyError: 'tst-linux64-ec2-300'
reconfig_1397206657.log:[buildbot-master103.srv.releng.scl3.mozilla.com] out: 	exceptions.KeyError: 'tst-linux64-ec2-300'
reconfig_1397206657.log:[buildbot-master104.srv.releng.scl3.mozilla.com] out: 	exceptions.KeyError: 'tst-linux64-ec2-300'
reconfig_1397206657.log:[buildbot-master105.srv.releng.scl3.mozilla.com] out: 	exceptions.KeyError: 'tst-linux64-ec2-300'

Comment 7

[[e-mail] (:bhearsum)]

Comment on attachment 8405297 [details] [diff] [review]
buildbot-configs patch

I still don't know why this happened, but I feel like it was a freak occurence. I just checked a couple of masters and this key definitely exists:
>>> master.botmaster.slaves['tst-linux64-ec2-300']
<BuildSlave 'tst-linux64-ec2-300'>
>>> 

I've backed this out. If it causes issues with the next reconfig I'll deal with any fallout.

Comment 8

[[e-mail] (:bhearsum)]

Broke things again, wtf

Comment 9

[[e-mail] (:bhearsum)]

I dunno what to do about this right now, but it's pretty low priority. We can probably fix it the next time we clean up slave lists.