638702 - Exception on adding new slaves to config

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Nick Thomas [:nthomas] (UTC+12)]

http://hg.mozilla.org/build/buildbot-configs/rev/3b5da1dd4f26 adds linux-ix-slave01 to the moz2 prod pool. On reconfig we consistently hit:

2011-03-04 00:03:36-0800 [-] Creating BuildMaster -- buildbot.version: 0.8.2-hg-a63f22816750-production-0.8
2011-03-04 00:03:36-0800 [-] loading configuration from /builds/buildbot/builder_master/master.cfg
2011-03-04 00:03:40-0800 [-] error during loadConfig
2011-03-04 00:03:40-0800 [-] Unhandled Error
        Traceback (most recent call last):
          File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/twisted/application/app.py", line 311, in runReactorWithLogging
            reactor.run()
          File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/twisted/internet/base.py", line 1165, in run
            self.mainLoop()
          File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/twisted/internet/base.py", line 1174, in mainLoop
            self.runUntilCurrent()
          File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/twisted/internet/base.py", line 796, in runUntilCurrent
            call.func(*call.args, **call.kw)
        --- <exception caught here> ---
          File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/master.py", line 600, in loadTheConfigFile
            d = self.loadConfig(f)
          File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/master.py", line 780, in loadConfig
            % (b['name'], n))
        exceptions.ValueError: builder Linux places build uses undefined slave linux-ix-slave01

That was while using fabric. Fixing this requires connecting to the master and doing a manual 'make reconfig'.

It's been a while since we added new slaves and the # of files we load to create the master.cfg has grown quite complicated, so I suspect something is no longer reconfig safe.

Comment 1

[[e-mail] (:bhearsum)]

We hit reconfig issues in the buildbot-configs patches in bug 572064, which uncovered an existing issue in those files. Maybe this will be fixed when that re-lands? Regardless, P3 =\.

Comment 2

[Lukas Blakk [:lsblakk] use ?needinfo]

Is this still happening? Did bug 572064 resolve the issue?

Comment 3

[[e-mail] (:bhearsum)]

I don't think this is an issue anymore, We've added tons of slaves in the past couple of years without issue.